2024-02-21T23:25:37.851 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-02-21T23:25:37.851 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2024-02-21_23:06:32-fs-wip-yuri10-testing-2024-02-21-1249-reef-distro-default-smithi/7570183 --name yuriw-2024-02-21_23:06:32-fs-wip-yuri10-testing-2024-02-21-1249-reef-distro-default-smithi --description fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} --verbose --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2024-02-21_23:06:32-fs-wip-yuri10-testing-2024-02-21-1249-reef-distro-default-smithi/7570183/orig.config.yaml 2024-02-21T23:25:37.906 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-21T23:25:37.993 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-02-21_23:06:32-fs-wip-yuri10-testing-2024-02-21-1249-reef-distro-default-smithi/7570183 branch: wip-yuri10-testing-2024-02-21-1249-reef description: fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} email: yweinste@redhat.com first_in_suite: false job_id: '7570183' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2024-02-21_23:06:32-fs-wip-yuri10-testing-2024-02-21-1249-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-yuri10-testing-2024-02-21-1249-reef ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 1 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - slow metadata IO - SLOW_OPS - slow request sha1: c1dc18f444603397768fcb7c34a8488b10284df6 subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: c1dc18f444603397768fcb7c34a8488b10284df6 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - wsync syntax: v2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri10-testing-2024-02-21-1249-reef sha1: c1dc18f444603397768fcb7c34a8488b10284df6 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 1691 sha1: c1dc18f444603397768fcb7c34a8488b10284df6 sleep_before_teardown: 0 subset: 111/120000 suite: fs suite_branch: wip-yuri10-testing-2024-02-21-1249-reef suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_c1dc18f444603397768fcb7c34a8488b10284df6/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: c1dc18f444603397768fcb7c34a8488b10284df6 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi090.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIsGbe8HkzhZE7QgpRJYvvS2/F3rk3+Sld7SbTDLMbrqSSFyoj3YfKfJ2KLL9DPsXS5F2GnhsB0d/msACzpTVlg= smithi176.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCoyhkEW5lZGHJcwJG3OfE/cNEzDU1Txk/PsJ5geG6e60QW55DpDVm/CbLAMmbRc/vh8k5rwNMjDeiHIQ5DMChg= smithi178.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM5c7YRZvTsR1p2rGWIPkA7BaA2hCTX8hzMGAddvN5CwxwIwIzSkhSOOO3cjWtq3RpFs7pOCzOII6Pb1ofvzKN0= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-02-21_23:06:32 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-21T23:25:37.994 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_c1dc18f444603397768fcb7c34a8488b10284df6/qa; will attempt to use it 2024-02-21T23:25:37.994 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_c1dc18f444603397768fcb7c34a8488b10284df6/qa/tasks 2024-02-21T23:25:37.995 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-21T23:25:37.996 INFO:teuthology.task.internal:Checking packages... 2024-02-21T23:25:38.022 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash 'c1dc18f444603397768fcb7c34a8488b10284df6' 2024-02-21T23:25:38.022 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-21T23:25:38.023 INFO:teuthology.packaging:ref: None 2024-02-21T23:25:38.023 INFO:teuthology.packaging:tag: None 2024-02-21T23:25:38.023 INFO:teuthology.packaging:branch: wip-yuri10-testing-2024-02-21-1249-reef 2024-02-21T23:25:38.023 INFO:teuthology.packaging:sha1: c1dc18f444603397768fcb7c34a8488b10284df6 2024-02-21T23:25:38.023 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri10-testing-2024-02-21-1249-reef 2024-02-21T23:25:38.158 INFO:teuthology.task.internal:Found packages for ceph version 18.2.1-638.gc1dc18f4 2024-02-21T23:25:38.160 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-21T23:25:38.168 INFO:teuthology.task.internal:no buildpackages task found 2024-02-21T23:25:38.168 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-21T23:25:38.212 INFO:teuthology.task.internal:Saving configuration 2024-02-21T23:25:38.234 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-21T23:25:38.241 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-21T23:25:38.267 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi090.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-21_23:06:32-fs-wip-yuri10-testing-2024-02-21-1249-reef-distro-default-smithi/7570183', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-21 23:19:43.991167', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIsGbe8HkzhZE7QgpRJYvvS2/F3rk3+Sld7SbTDLMbrqSSFyoj3YfKfJ2KLL9DPsXS5F2GnhsB0d/msACzpTVlg='} 2024-02-21T23:25:38.287 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi176.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-21_23:06:32-fs-wip-yuri10-testing-2024-02-21-1249-reef-distro-default-smithi/7570183', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-21 23:19:43.988364', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCoyhkEW5lZGHJcwJG3OfE/cNEzDU1Txk/PsJ5geG6e60QW55DpDVm/CbLAMmbRc/vh8k5rwNMjDeiHIQ5DMChg='} 2024-02-21T23:25:38.311 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi178.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-21_23:06:32-fs-wip-yuri10-testing-2024-02-21-1249-reef-distro-default-smithi/7570183', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-21 23:19:43.990009', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM5c7YRZvTsR1p2rGWIPkA7BaA2hCTX8hzMGAddvN5CwxwIwIzSkhSOOO3cjWtq3RpFs7pOCzOII6Pb1ofvzKN0='} 2024-02-21T23:25:38.311 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-21T23:25:38.318 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-02-21T23:25:38.318 INFO:teuthology.task.internal:roles: ubuntu@smithi176.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-02-21T23:25:38.318 INFO:teuthology.task.internal:roles: ubuntu@smithi178.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-02-21T23:25:38.318 INFO:teuthology.run_tasks:Running task console_log... 2024-02-21T23:25:38.440 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd82ba7b160>, signals=[15]) 2024-02-21T23:25:38.440 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-21T23:25:38.448 INFO:teuthology.task.internal:Opening connections... 2024-02-21T23:25:38.448 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi090.front.sepia.ceph.com 2024-02-21T23:25:38.451 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-21T23:25:38.537 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi176.front.sepia.ceph.com 2024-02-21T23:25:38.538 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-21T23:25:38.610 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi178.front.sepia.ceph.com 2024-02-21T23:25:38.611 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-21T23:25:38.682 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-21T23:25:38.693 DEBUG:teuthology.orchestra.run.smithi090:> uname -m 2024-02-21T23:25:38.713 INFO:teuthology.orchestra.run.smithi090.stdout:x86_64 2024-02-21T23:25:38.714 DEBUG:teuthology.orchestra.run.smithi090:> cat /etc/os-release 2024-02-21T23:25:38.773 INFO:teuthology.orchestra.run.smithi090.stdout:NAME="Red Hat Enterprise Linux" 2024-02-21T23:25:38.773 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION="8.6 (Ootpa)" 2024-02-21T23:25:38.773 INFO:teuthology.orchestra.run.smithi090.stdout:ID="rhel" 2024-02-21T23:25:38.773 INFO:teuthology.orchestra.run.smithi090.stdout:ID_LIKE="fedora" 2024-02-21T23:25:38.774 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION_ID="8.6" 2024-02-21T23:25:38.774 INFO:teuthology.orchestra.run.smithi090.stdout:PLATFORM_ID="platform:el8" 2024-02-21T23:25:38.774 INFO:teuthology.orchestra.run.smithi090.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-02-21T23:25:38.774 INFO:teuthology.orchestra.run.smithi090.stdout:ANSI_COLOR="0;31" 2024-02-21T23:25:38.774 INFO:teuthology.orchestra.run.smithi090.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-02-21T23:25:38.774 INFO:teuthology.orchestra.run.smithi090.stdout:HOME_URL="https://www.redhat.com/" 2024-02-21T23:25:38.774 INFO:teuthology.orchestra.run.smithi090.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-02-21T23:25:38.774 INFO:teuthology.orchestra.run.smithi090.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-21T23:25:38.774 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:25:38.774 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-21T23:25:38.774 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-02-21T23:25:38.774 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-02-21T23:25:38.774 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-02-21T23:25:38.775 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server 2024-02-21T23:25:38.795 DEBUG:teuthology.orchestra.run.smithi176:> uname -m 2024-02-21T23:25:38.816 INFO:teuthology.orchestra.run.smithi176.stdout:x86_64 2024-02-21T23:25:38.817 DEBUG:teuthology.orchestra.run.smithi176:> cat /etc/os-release 2024-02-21T23:25:38.876 INFO:teuthology.orchestra.run.smithi176.stdout:NAME="Red Hat Enterprise Linux" 2024-02-21T23:25:38.876 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION="8.6 (Ootpa)" 2024-02-21T23:25:38.876 INFO:teuthology.orchestra.run.smithi176.stdout:ID="rhel" 2024-02-21T23:25:38.877 INFO:teuthology.orchestra.run.smithi176.stdout:ID_LIKE="fedora" 2024-02-21T23:25:38.877 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION_ID="8.6" 2024-02-21T23:25:38.877 INFO:teuthology.orchestra.run.smithi176.stdout:PLATFORM_ID="platform:el8" 2024-02-21T23:25:38.877 INFO:teuthology.orchestra.run.smithi176.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-02-21T23:25:38.877 INFO:teuthology.orchestra.run.smithi176.stdout:ANSI_COLOR="0;31" 2024-02-21T23:25:38.877 INFO:teuthology.orchestra.run.smithi176.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-02-21T23:25:38.877 INFO:teuthology.orchestra.run.smithi176.stdout:HOME_URL="https://www.redhat.com/" 2024-02-21T23:25:38.877 INFO:teuthology.orchestra.run.smithi176.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-02-21T23:25:38.877 INFO:teuthology.orchestra.run.smithi176.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-21T23:25:38.877 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-21T23:25:38.877 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-21T23:25:38.877 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-02-21T23:25:38.877 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-02-21T23:25:38.877 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-02-21T23:25:38.878 INFO:teuthology.lock.ops:Updating smithi176.front.sepia.ceph.com on lock server 2024-02-21T23:25:38.895 DEBUG:teuthology.orchestra.run.smithi178:> uname -m 2024-02-21T23:25:38.916 INFO:teuthology.orchestra.run.smithi178.stdout:x86_64 2024-02-21T23:25:38.916 DEBUG:teuthology.orchestra.run.smithi178:> cat /etc/os-release 2024-02-21T23:25:38.975 INFO:teuthology.orchestra.run.smithi178.stdout:NAME="Red Hat Enterprise Linux" 2024-02-21T23:25:38.976 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION="8.6 (Ootpa)" 2024-02-21T23:25:38.976 INFO:teuthology.orchestra.run.smithi178.stdout:ID="rhel" 2024-02-21T23:25:38.976 INFO:teuthology.orchestra.run.smithi178.stdout:ID_LIKE="fedora" 2024-02-21T23:25:38.976 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION_ID="8.6" 2024-02-21T23:25:38.976 INFO:teuthology.orchestra.run.smithi178.stdout:PLATFORM_ID="platform:el8" 2024-02-21T23:25:38.976 INFO:teuthology.orchestra.run.smithi178.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-02-21T23:25:38.976 INFO:teuthology.orchestra.run.smithi178.stdout:ANSI_COLOR="0;31" 2024-02-21T23:25:38.976 INFO:teuthology.orchestra.run.smithi178.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-02-21T23:25:38.976 INFO:teuthology.orchestra.run.smithi178.stdout:HOME_URL="https://www.redhat.com/" 2024-02-21T23:25:38.976 INFO:teuthology.orchestra.run.smithi178.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-02-21T23:25:38.976 INFO:teuthology.orchestra.run.smithi178.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-21T23:25:38.976 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-21T23:25:38.977 INFO:teuthology.orchestra.run.smithi178.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-21T23:25:38.977 INFO:teuthology.orchestra.run.smithi178.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-02-21T23:25:38.977 INFO:teuthology.orchestra.run.smithi178.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-02-21T23:25:38.977 INFO:teuthology.orchestra.run.smithi178.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-02-21T23:25:38.977 INFO:teuthology.lock.ops:Updating smithi178.front.sepia.ceph.com on lock server 2024-02-21T23:25:38.995 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-21T23:25:39.008 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-21T23:25:39.033 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-21T23:25:39.034 DEBUG:teuthology.orchestra.run.smithi090:> test '!' -e /home/ubuntu/cephtest 2024-02-21T23:25:39.036 DEBUG:teuthology.orchestra.run.smithi176:> test '!' -e /home/ubuntu/cephtest 2024-02-21T23:25:39.039 DEBUG:teuthology.orchestra.run.smithi178:> test '!' -e /home/ubuntu/cephtest 2024-02-21T23:25:39.057 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-21T23:25:39.065 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-21T23:25:39.065 DEBUG:teuthology.orchestra.run.smithi090:> test -z $(ls -A /var/lib/ceph) 2024-02-21T23:25:39.096 DEBUG:teuthology.orchestra.run.smithi176:> test -z $(ls -A /var/lib/ceph) 2024-02-21T23:25:39.098 DEBUG:teuthology.orchestra.run.smithi178:> test -z $(ls -A /var/lib/ceph) 2024-02-21T23:25:39.124 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-21T23:25:39.213 INFO:teuthology.run_tasks:Running task kernel... 2024-02-21T23:25:39.233 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-02-21T23:25:39.233 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-02-21T23:25:39.233 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-02-21T23:25:39.234 DEBUG:teuthology.orchestra.run.smithi090:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-21T23:25:39.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-21T23:25:39.252 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2024-02-21T23:25:39.312 INFO:teuthology.orchestra.run.smithi090.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-02-21T23:25:39.313 INFO:teuthology.task.kernel:Running kernel on smithi090: 4.18.0-513.9.1.el8_9.x86_64 2024-02-21T23:25:39.313 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum install -y kernel 2024-02-21T23:25:41.253 INFO:teuthology.orchestra.run.smithi090.stdout:Updating Subscription Management repositories. 2024-02-21T23:25:41.254 INFO:teuthology.orchestra.run.smithi090.stdout:Unable to read consumer identity 2024-02-21T23:25:41.419 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:25:41.419 INFO:teuthology.orchestra.run.smithi090.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-02-21T23:25:41.420 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:25:44.006 INFO:teuthology.orchestra.run.smithi090.stdout:Copr repo for python3-asyncssh owned by ceph 14 kB/s | 3.5 kB 00:00 2024-02-21T23:25:45.057 INFO:teuthology.orchestra.run.smithi090.stdout:Extra Packages for Enterprise Linux 15 MB/s | 16 MB 00:01 2024-02-21T23:25:48.583 INFO:teuthology.orchestra.run.smithi090.stdout:lab-extras 473 kB/s | 24 kB 00:00 2024-02-21T23:25:49.310 INFO:teuthology.orchestra.run.smithi090.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-02-21T23:25:49.310 INFO:teuthology.orchestra.run.smithi090.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-02-21T23:25:49.395 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-21T23:25:49.396 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-02-21T23:25:49.396 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-21T23:25:49.534 DEBUG:teuthology.orchestra.run.smithi090:> echo no | sudo yum reinstall kernel || true 2024-02-21T23:25:49.872 INFO:teuthology.orchestra.run.smithi090.stdout:Updating Subscription Management repositories. 2024-02-21T23:25:49.873 INFO:teuthology.orchestra.run.smithi090.stdout:Unable to read consumer identity 2024-02-21T23:25:50.125 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:25:50.126 INFO:teuthology.orchestra.run.smithi090.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-02-21T23:25:50.126 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:25:50.437 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:02 ago on Wed 21 Feb 2024 11:25:48 PM UTC. 2024-02-21T23:25:50.515 INFO:teuthology.orchestra.run.smithi090.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-02-21T23:25:50.524 INFO:teuthology.orchestra.run.smithi090.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-02-21T23:25:50.626 INFO:teuthology.orchestra.run.smithi090.stderr:Error: No packages marked for reinstall. 2024-02-21T23:25:50.673 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum reinstall -y kernel || true 2024-02-21T23:25:51.008 INFO:teuthology.orchestra.run.smithi090.stdout:Updating Subscription Management repositories. 2024-02-21T23:25:51.009 INFO:teuthology.orchestra.run.smithi090.stdout:Unable to read consumer identity 2024-02-21T23:25:51.231 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:25:51.231 INFO:teuthology.orchestra.run.smithi090.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-02-21T23:25:51.231 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:25:51.535 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:03 ago on Wed 21 Feb 2024 11:25:48 PM UTC. 2024-02-21T23:25:51.598 INFO:teuthology.orchestra.run.smithi090.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-02-21T23:25:51.598 INFO:teuthology.orchestra.run.smithi090.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-02-21T23:25:51.636 INFO:teuthology.orchestra.run.smithi090.stderr:Error: No packages marked for reinstall. 2024-02-21T23:25:51.683 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q kernel | sort -rV | head -n 1 2024-02-21T23:25:51.861 INFO:teuthology.orchestra.run.smithi090.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-02-21T23:25:51.861 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-02-21T23:25:51.861 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-02-21T23:25:51.861 DEBUG:teuthology.parallel:result is None 2024-02-21T23:25:51.862 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-21T23:25:51.869 INFO:teuthology.task.internal:Creating test directory... 2024-02-21T23:25:51.870 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-21T23:25:51.872 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-21T23:25:51.874 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-21T23:25:51.895 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-21T23:25:51.930 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-21T23:25:51.953 INFO:teuthology.task.internal:Creating archive directory... 2024-02-21T23:25:51.954 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-21T23:25:51.956 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-21T23:25:51.958 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-21T23:25:51.997 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-21T23:25:52.005 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-21T23:25:52.005 DEBUG:teuthology.orchestra.run.smithi176:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-21T23:25:52.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-21T23:25:52.043 DEBUG:teuthology.orchestra.run.smithi178:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-21T23:25:52.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-21T23:25:52.061 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-21T23:25:52.063 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-21T23:25:52.085 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-21T23:25:52.097 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-21T23:25:52.108 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-21T23:25:52.121 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-21T23:25:52.140 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-21T23:25:52.146 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-21T23:25:52.163 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-21T23:25:52.164 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-21T23:25:52.171 INFO:teuthology.task.internal:Configuring sudo... 2024-02-21T23:25:52.172 DEBUG:teuthology.orchestra.run.smithi090:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-21T23:25:52.174 DEBUG:teuthology.orchestra.run.smithi176:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-21T23:25:52.184 DEBUG:teuthology.orchestra.run.smithi178:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-21T23:25:52.213 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-21T23:25:52.222 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-21T23:25:52.223 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-21T23:25:52.241 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-21T23:25:52.253 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-21T23:25:52.271 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-21T23:25:52.339 DEBUG:teuthology.orchestra.run.smithi090:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-21T23:25:52.424 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-21T23:25:52.464 DEBUG:teuthology.orchestra.run.smithi090:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-21T23:25:52.508 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-21T23:25:52.509 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-21T23:25:52.575 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-21T23:25:52.624 DEBUG:teuthology.orchestra.run.smithi176:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-21T23:25:52.675 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-21T23:25:52.761 DEBUG:teuthology.orchestra.run.smithi176:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-21T23:25:52.803 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-02-21T23:25:52.804 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-21T23:25:52.871 DEBUG:teuthology.orchestra.run.smithi178:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-21T23:25:52.916 DEBUG:teuthology.orchestra.run.smithi178:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-21T23:25:52.971 DEBUG:teuthology.orchestra.run.smithi178:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-21T23:25:53.057 DEBUG:teuthology.orchestra.run.smithi178:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-21T23:25:53.101 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-02-21T23:25:53.101 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-21T23:25:53.169 DEBUG:teuthology.orchestra.run.smithi090:> sudo service rsyslog restart 2024-02-21T23:25:53.171 DEBUG:teuthology.orchestra.run.smithi176:> sudo service rsyslog restart 2024-02-21T23:25:53.174 DEBUG:teuthology.orchestra.run.smithi178:> sudo service rsyslog restart 2024-02-21T23:25:53.223 INFO:teuthology.orchestra.run.smithi176.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-21T23:25:53.233 INFO:teuthology.orchestra.run.smithi090.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-21T23:25:53.264 INFO:teuthology.orchestra.run.smithi178.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-21T23:25:53.687 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-21T23:25:53.698 INFO:teuthology.task.internal:Starting timer... 2024-02-21T23:25:53.698 INFO:teuthology.run_tasks:Running task pcp... 2024-02-21T23:25:53.757 INFO:teuthology.run_tasks:Running task selinux... 2024-02-21T23:25:53.780 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']} 2024-02-21T23:25:53.781 DEBUG:teuthology.orchestra.run.smithi090:> sudo service auditd rotate 2024-02-21T23:25:53.882 INFO:teuthology.orchestra.run.smithi090.stdout:Rotating logs: 2024-02-21T23:25:53.884 DEBUG:teuthology.orchestra.run.smithi176:> sudo service auditd rotate 2024-02-21T23:25:53.956 INFO:teuthology.orchestra.run.smithi176.stdout:Rotating logs: 2024-02-21T23:25:53.958 DEBUG:teuthology.orchestra.run.smithi178:> sudo service auditd rotate 2024-02-21T23:25:54.039 INFO:teuthology.orchestra.run.smithi178.stdout:Rotating logs: 2024-02-21T23:25:54.041 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-21T23:25:54.041 DEBUG:teuthology.orchestra.run.smithi090:> /usr/sbin/getenforce 2024-02-21T23:25:54.066 INFO:teuthology.orchestra.run.smithi090.stdout:Permissive 2024-02-21T23:25:54.066 DEBUG:teuthology.orchestra.run.smithi176:> /usr/sbin/getenforce 2024-02-21T23:25:54.086 INFO:teuthology.orchestra.run.smithi176.stdout:Permissive 2024-02-21T23:25:54.086 DEBUG:teuthology.orchestra.run.smithi178:> /usr/sbin/getenforce 2024-02-21T23:25:54.120 INFO:teuthology.orchestra.run.smithi178.stdout:Permissive 2024-02-21T23:25:54.121 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi090.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi176.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi178.front.sepia.ceph.com': 'permissive'} 2024-02-21T23:25:54.121 DEBUG:teuthology.orchestra.run.smithi090:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-21T23:25:54.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-21T23:25:54.149 DEBUG:teuthology.orchestra.run.smithi176:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-21T23:25:54.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-21T23:25:54.176 DEBUG:teuthology.orchestra.run.smithi178:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-21T23:25:54.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-21T23:25:54.204 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-21T23:25:54.204 DEBUG:teuthology.orchestra.run.smithi090:> sudo /usr/sbin/setenforce permissive 2024-02-21T23:25:54.230 DEBUG:teuthology.orchestra.run.smithi176:> sudo /usr/sbin/setenforce permissive 2024-02-21T23:25:54.257 DEBUG:teuthology.orchestra.run.smithi178:> sudo /usr/sbin/setenforce permissive 2024-02-21T23:25:54.285 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-21T23:25:54.297 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-02-21T23:25:54.303 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-02-21T23:25:54.439 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-21T23:25:54.454 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-21T23:25:54.455 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi090.front.sepia.ceph.com,smithi176.front.sepia.ceph.com,smithi178.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-21T23:35:09.376 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi090.front.sepia.ceph.com'), Remote(name='ubuntu@smithi176.front.sepia.ceph.com'), Remote(name='ubuntu@smithi178.front.sepia.ceph.com')] 2024-02-21T23:35:09.377 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-21T23:35:09.378 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-21T23:35:09.453 DEBUG:teuthology.orchestra.run.smithi090:> true 2024-02-21T23:35:09.525 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-21T23:35:09.525 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-02-21T23:35:09.526 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-21T23:35:09.600 DEBUG:teuthology.orchestra.run.smithi176:> true 2024-02-21T23:35:09.670 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-02-21T23:35:09.670 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-02-21T23:35:09.671 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-21T23:35:09.743 DEBUG:teuthology.orchestra.run.smithi178:> true 2024-02-21T23:35:09.814 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-02-21T23:35:09.815 INFO:teuthology.run_tasks:Running task clock... 2024-02-21T23:35:09.829 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-21T23:35:09.830 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-21T23:35:09.830 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-21T23:35:09.832 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-21T23:35:09.832 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-21T23:35:09.834 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-21T23:35:09.835 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-21T23:35:09.862 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-21T23:35:09.866 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-21T23:35:09.877 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-21T23:35:09.882 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-21T23:35:09.889 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-21T23:35:09.904 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-21T23:35:09.946 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: ntpd: command not found 2024-02-21T23:35:09.957 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: ntpd: command not found 2024-02-21T23:35:09.958 INFO:teuthology.orchestra.run.smithi090.stdout:506 Cannot talk to daemon 2024-02-21T23:35:09.970 INFO:teuthology.orchestra.run.smithi176.stdout:506 Cannot talk to daemon 2024-02-21T23:35:09.973 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-21T23:35:09.976 INFO:teuthology.orchestra.run.smithi178.stderr:sudo: ntpd: command not found 2024-02-21T23:35:09.984 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-21T23:35:09.987 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-21T23:35:09.988 INFO:teuthology.orchestra.run.smithi178.stdout:506 Cannot talk to daemon 2024-02-21T23:35:09.998 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-21T23:35:10.003 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-21T23:35:10.017 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-21T23:35:10.054 INFO:teuthology.orchestra.run.smithi090.stderr:bash: ntpq: command not found 2024-02-21T23:35:10.057 INFO:teuthology.orchestra.run.smithi090.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-21T23:35:10.058 INFO:teuthology.orchestra.run.smithi090.stdout:=============================================================================== 2024-02-21T23:35:10.058 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-21T23:35:10.058 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-21T23:35:10.058 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-21T23:35:10.058 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-21T23:35:10.079 INFO:teuthology.orchestra.run.smithi176.stderr:bash: ntpq: command not found 2024-02-21T23:35:10.084 INFO:teuthology.orchestra.run.smithi176.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-21T23:35:10.084 INFO:teuthology.orchestra.run.smithi176.stdout:=============================================================================== 2024-02-21T23:35:10.084 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-21T23:35:10.084 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-21T23:35:10.084 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-21T23:35:10.084 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-21T23:35:10.100 INFO:teuthology.orchestra.run.smithi178.stderr:bash: ntpq: command not found 2024-02-21T23:35:10.104 INFO:teuthology.orchestra.run.smithi178.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-21T23:35:10.104 INFO:teuthology.orchestra.run.smithi178.stdout:=============================================================================== 2024-02-21T23:35:10.104 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-21T23:35:10.104 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-21T23:35:10.104 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-21T23:35:10.104 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-21T23:35:10.105 INFO:teuthology.run_tasks:Running task pexec... 2024-02-21T23:35:10.115 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-21T23:35:10.115 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi090.front.sepia.ceph.com 2024-02-21T23:35:10.116 DEBUG:teuthology.orchestra.run.smithi090:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-21T23:35:10.116 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi176.front.sepia.ceph.com 2024-02-21T23:35:10.116 DEBUG:teuthology.orchestra.run.smithi176:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-21T23:35:10.116 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi178.front.sepia.ceph.com 2024-02-21T23:35:10.117 DEBUG:teuthology.orchestra.run.smithi178:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-21T23:35:10.492 INFO:teuthology.orchestra.run.smithi090.stdout:Updating Subscription Management repositories. 2024-02-21T23:35:10.502 INFO:teuthology.orchestra.run.smithi176.stdout:Updating Subscription Management repositories. 2024-02-21T23:35:10.523 INFO:teuthology.orchestra.run.smithi178.stdout:Updating Subscription Management repositories. 2024-02-21T23:35:12.334 INFO:teuthology.orchestra.run.smithi090.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 36 kB/s | 4.1 kB 00:00 2024-02-21T23:35:12.435 INFO:teuthology.orchestra.run.smithi178.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 35 kB/s | 4.5 kB 00:00 2024-02-21T23:35:12.523 INFO:teuthology.orchestra.run.smithi176.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 24 kB/s | 4.5 kB 00:00 2024-02-21T23:35:12.674 INFO:teuthology.orchestra.run.smithi090.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 35 kB/s | 4.5 kB 00:00 2024-02-21T23:35:13.100 INFO:teuthology.orchestra.run.smithi178.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 38 kB/s | 4.1 kB 00:00 2024-02-21T23:35:13.148 INFO:teuthology.orchestra.run.smithi176.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 43 kB/s | 4.1 kB 00:00 2024-02-21T23:35:15.548 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-21T23:35:15.549 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-21T23:35:15.549 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-02-21T23:35:15.549 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-21T23:35:15.549 INFO:teuthology.orchestra.run.smithi090.stdout:Resetting modules: 2024-02-21T23:35:15.550 INFO:teuthology.orchestra.run.smithi090.stdout: container-tools 2024-02-21T23:35:15.550 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:35:15.550 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-21T23:35:15.550 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-21T23:35:15.550 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:35:16.112 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-02-21T23:35:16.113 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-21T23:35:16.113 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2024-02-21T23:35:16.113 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-21T23:35:16.113 INFO:teuthology.orchestra.run.smithi178.stdout:Resetting modules: 2024-02-21T23:35:16.113 INFO:teuthology.orchestra.run.smithi178.stdout: container-tools 2024-02-21T23:35:16.113 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-21T23:35:16.113 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-02-21T23:35:16.113 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-21T23:35:16.113 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-21T23:35:16.134 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-02-21T23:35:16.135 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-21T23:35:16.135 INFO:teuthology.orchestra.run.smithi176.stdout: Package Architecture Version Repository Size 2024-02-21T23:35:16.135 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-21T23:35:16.135 INFO:teuthology.orchestra.run.smithi176.stdout:Resetting modules: 2024-02-21T23:35:16.135 INFO:teuthology.orchestra.run.smithi176.stdout: container-tools 2024-02-21T23:35:16.135 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-21T23:35:16.135 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-02-21T23:35:16.135 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-21T23:35:16.135 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-21T23:35:23.368 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-21T23:35:23.838 INFO:teuthology.orchestra.run.smithi090.stdout:Updating Subscription Management repositories. 2024-02-21T23:35:25.664 INFO:teuthology.orchestra.run.smithi090.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 41 kB/s | 4.1 kB 00:00 2024-02-21T23:35:26.001 INFO:teuthology.orchestra.run.smithi090.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 36 kB/s | 4.5 kB 00:00 2024-02-21T23:35:26.529 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-02-21T23:35:27.028 INFO:teuthology.orchestra.run.smithi178.stdout:Updating Subscription Management repositories. 2024-02-21T23:35:28.315 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-02-21T23:35:28.792 INFO:teuthology.orchestra.run.smithi176.stdout:Updating Subscription Management repositories. 2024-02-21T23:35:28.817 INFO:teuthology.orchestra.run.smithi178.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 27 kB/s | 4.5 kB 00:00 2024-02-21T23:35:29.033 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-21T23:35:29.191 INFO:teuthology.orchestra.run.smithi090.stdout:========================================================================================================== 2024-02-21T23:35:29.191 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-02-21T23:35:29.191 INFO:teuthology.orchestra.run.smithi090.stdout:========================================================================================================== 2024-02-21T23:35:29.191 INFO:teuthology.orchestra.run.smithi090.stdout:Installing group/module packages: 2024-02-21T23:35:29.191 INFO:teuthology.orchestra.run.smithi090.stdout: aardvark-dns x86_64 2:1.7.0-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 1.0 M 2024-02-21T23:35:29.191 INFO:teuthology.orchestra.run.smithi090.stdout: buildah x86_64 1:1.31.3-3.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 8.8 M 2024-02-21T23:35:29.191 INFO:teuthology.orchestra.run.smithi090.stdout: cockpit-podman noarch 75-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 739 k 2024-02-21T23:35:29.191 INFO:teuthology.orchestra.run.smithi090.stdout: crun x86_64 1.8.7-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 239 k 2024-02-21T23:35:29.191 INFO:teuthology.orchestra.run.smithi090.stdout: netavark x86_64 2:1.7.0-2.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-02-21T23:35:29.191 INFO:teuthology.orchestra.run.smithi090.stdout: python3-podman noarch 4.6.0-2.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 154 k 2024-02-21T23:35:29.191 INFO:teuthology.orchestra.run.smithi090.stdout: skopeo x86_64 2:1.13.3-3.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 8.1 M 2024-02-21T23:35:29.191 INFO:teuthology.orchestra.run.smithi090.stdout: toolbox x86_64 0.0.99.4-5.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 2.5 M 2024-02-21T23:35:29.192 INFO:teuthology.orchestra.run.smithi090.stdout: udica noarch 0.2.6-20.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 49 k 2024-02-21T23:35:29.192 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-02-21T23:35:29.192 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2024-02-21T23:35:29.192 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2024-02-21T23:35:29.192 INFO:teuthology.orchestra.run.smithi090.stdout: yajl x86_64 2.1.0-12.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2024-02-21T23:35:29.192 INFO:teuthology.orchestra.run.smithi090.stdout:Installing weak dependencies: 2024-02-21T23:35:29.192 INFO:teuthology.orchestra.run.smithi090.stdout: criu-libs x86_64 3.18-4.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 39 k 2024-02-21T23:35:29.192 INFO:teuthology.orchestra.run.smithi090.stdout:Installing module profiles: 2024-02-21T23:35:29.192 INFO:teuthology.orchestra.run.smithi090.stdout: container-tools/common 2024-02-21T23:35:29.192 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:35:29.192 INFO:teuthology.orchestra.run.smithi090.stdout:Enabling module streams: 2024-02-21T23:35:29.192 INFO:teuthology.orchestra.run.smithi090.stdout: container-tools rhel8 2024-02-21T23:35:29.192 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:35:29.192 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-21T23:35:29.192 INFO:teuthology.orchestra.run.smithi090.stdout:========================================================================================================== 2024-02-21T23:35:29.193 INFO:teuthology.orchestra.run.smithi090.stdout:Install 13 Packages 2024-02-21T23:35:29.193 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:35:29.193 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 26 M 2024-02-21T23:35:29.193 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 87 M 2024-02-21T23:35:29.193 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-21T23:35:29.341 INFO:teuthology.orchestra.run.smithi090.stdout:(1/13): python3-pyxdg-0.25-16.el8.noarch.rpm 638 kB/s | 94 kB 00:00 2024-02-21T23:35:29.366 INFO:teuthology.orchestra.run.smithi090.stdout:(2/13): python3-pytoml-0.1.14-5.git7dea353.el8. 144 kB/s | 25 kB 00:00 2024-02-21T23:35:29.408 INFO:teuthology.orchestra.run.smithi090.stdout:(3/13): criu-libs-3.18-4.module+el8.9.0+20326+3 184 kB/s | 39 kB 00:00 2024-02-21T23:35:29.475 INFO:teuthology.orchestra.run.smithi090.stdout:(4/13): yajl-2.1.0-12.el8.x86_64.rpm 305 kB/s | 41 kB 00:00 2024-02-21T23:35:29.542 INFO:teuthology.orchestra.run.smithi090.stdout:(5/13): aardvark-dns-1.7.0-1.module+el8.9.0+212 5.7 MB/s | 1.0 MB 00:00 2024-02-21T23:35:29.552 INFO:teuthology.orchestra.run.smithi178.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 20 kB/s | 4.1 kB 00:00 2024-02-21T23:35:29.609 INFO:teuthology.orchestra.run.smithi090.stdout:(6/13): cockpit-podman-75-1.module+el8.9.0+2124 5.4 MB/s | 739 kB 00:00 2024-02-21T23:35:29.776 INFO:teuthology.orchestra.run.smithi090.stdout:(7/13): buildah-1.31.3-3.module+el8.9.0+21243+a 24 MB/s | 8.8 MB 00:00 2024-02-21T23:35:29.843 INFO:teuthology.orchestra.run.smithi090.stdout:(8/13): crun-1.8.7-1.module+el8.9.0+21243+a5865 796 kB/s | 239 kB 00:00 2024-02-21T23:35:29.927 INFO:teuthology.orchestra.run.smithi090.stdout:(9/13): netavark-1.7.0-2.module+el8.9.0+21243+a 12 MB/s | 3.7 MB 00:00 2024-02-21T23:35:29.952 INFO:teuthology.orchestra.run.smithi090.stdout:(10/13): python3-podman-4.6.0-2.module+el8.9.0+ 875 kB/s | 154 kB 00:00 2024-02-21T23:35:30.136 INFO:teuthology.orchestra.run.smithi090.stdout:(11/13): skopeo-1.13.3-3.module+el8.9.0+21243+a 28 MB/s | 8.1 MB 00:00 2024-02-21T23:35:30.188 INFO:teuthology.orchestra.run.smithi090.stdout:(12/13): udica-0.2.6-20.module+el8.9.0+21243+a5 209 kB/s | 49 kB 00:00 2024-02-21T23:35:30.261 INFO:teuthology.orchestra.run.smithi090.stdout:(13/13): toolbox-0.0.99.4-5.module+el8.9.0+2124 7.6 MB/s | 2.5 MB 00:00 2024-02-21T23:35:30.262 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-21T23:35:30.262 INFO:teuthology.orchestra.run.smithi090.stdout:Total 24 MB/s | 26 MB 00:01 2024-02-21T23:35:30.546 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-21T23:35:30.568 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-21T23:35:30.568 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-21T23:35:30.568 INFO:teuthology.orchestra.run.smithi176.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 32 kB/s | 4.5 kB 00:00 2024-02-21T23:35:30.799 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-21T23:35:30.807 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-21T23:35:31.193 INFO:teuthology.orchestra.run.smithi176.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 42 kB/s | 4.1 kB 00:00 2024-02-21T23:35:31.506 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-21T23:35:32.054 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a58653 1/13 2024-02-21T23:35:32.253 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x 2/13 2024-02-21T23:35:32.282 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-02-21T23:35:32.437 INFO:teuthology.orchestra.run.smithi178.stdout:========================================================================================================== 2024-02-21T23:35:32.437 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-02-21T23:35:32.437 INFO:teuthology.orchestra.run.smithi178.stdout:========================================================================================================== 2024-02-21T23:35:32.438 INFO:teuthology.orchestra.run.smithi178.stdout:Installing group/module packages: 2024-02-21T23:35:32.438 INFO:teuthology.orchestra.run.smithi178.stdout: aardvark-dns x86_64 2:1.7.0-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 1.0 M 2024-02-21T23:35:32.438 INFO:teuthology.orchestra.run.smithi178.stdout: buildah x86_64 1:1.31.3-3.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 8.8 M 2024-02-21T23:35:32.438 INFO:teuthology.orchestra.run.smithi178.stdout: cockpit-podman noarch 75-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 739 k 2024-02-21T23:35:32.438 INFO:teuthology.orchestra.run.smithi178.stdout: crun x86_64 1.8.7-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 239 k 2024-02-21T23:35:32.438 INFO:teuthology.orchestra.run.smithi178.stdout: netavark x86_64 2:1.7.0-2.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-02-21T23:35:32.438 INFO:teuthology.orchestra.run.smithi178.stdout: python3-podman noarch 4.6.0-2.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 154 k 2024-02-21T23:35:32.438 INFO:teuthology.orchestra.run.smithi178.stdout: skopeo x86_64 2:1.13.3-3.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 8.1 M 2024-02-21T23:35:32.438 INFO:teuthology.orchestra.run.smithi178.stdout: toolbox x86_64 0.0.99.4-5.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 2.5 M 2024-02-21T23:35:32.438 INFO:teuthology.orchestra.run.smithi178.stdout: udica noarch 0.2.6-20.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 49 k 2024-02-21T23:35:32.438 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-02-21T23:35:32.438 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2024-02-21T23:35:32.439 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2024-02-21T23:35:32.439 INFO:teuthology.orchestra.run.smithi178.stdout: yajl x86_64 2.1.0-12.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2024-02-21T23:35:32.439 INFO:teuthology.orchestra.run.smithi178.stdout:Installing weak dependencies: 2024-02-21T23:35:32.439 INFO:teuthology.orchestra.run.smithi178.stdout: criu-libs x86_64 3.18-4.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 39 k 2024-02-21T23:35:32.439 INFO:teuthology.orchestra.run.smithi178.stdout:Installing module profiles: 2024-02-21T23:35:32.439 INFO:teuthology.orchestra.run.smithi178.stdout: container-tools/common 2024-02-21T23:35:32.439 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-21T23:35:32.439 INFO:teuthology.orchestra.run.smithi178.stdout:Enabling module streams: 2024-02-21T23:35:32.439 INFO:teuthology.orchestra.run.smithi178.stdout: container-tools rhel8 2024-02-21T23:35:32.439 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-21T23:35:32.439 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-02-21T23:35:32.439 INFO:teuthology.orchestra.run.smithi178.stdout:========================================================================================================== 2024-02-21T23:35:32.439 INFO:teuthology.orchestra.run.smithi178.stdout:Install 13 Packages 2024-02-21T23:35:32.439 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-21T23:35:32.440 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 26 M 2024-02-21T23:35:32.440 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 87 M 2024-02-21T23:35:32.440 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-02-21T23:35:32.444 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : yajl-2.1.0-12.el8.x86_64 3/13 2024-02-21T23:35:32.605 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 4/13 2024-02-21T23:35:32.613 INFO:teuthology.orchestra.run.smithi178.stdout:(1/13): python3-pyxdg-0.25-16.el8.noarch.rpm 543 kB/s | 94 kB 00:00 2024-02-21T23:35:32.662 INFO:teuthology.orchestra.run.smithi178.stdout:(2/13): python3-pytoml-0.1.14-5.git7dea353.el8. 112 kB/s | 25 kB 00:00 2024-02-21T23:35:32.716 INFO:teuthology.orchestra.run.smithi178.stdout:(3/13): criu-libs-3.18-4.module+el8.9.0+20326+3 143 kB/s | 39 kB 00:00 2024-02-21T23:35:32.773 INFO:teuthology.orchestra.run.smithi178.stdout:(4/13): yajl-2.1.0-12.el8.x86_64.rpm 258 kB/s | 41 kB 00:00 2024-02-21T23:35:32.808 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 5/13 2024-02-21T23:35:32.832 INFO:teuthology.orchestra.run.smithi178.stdout:(5/13): aardvark-dns-1.7.0-1.module+el8.9.0+212 6.0 MB/s | 1.0 MB 00:00 2024-02-21T23:35:32.957 INFO:teuthology.orchestra.run.smithi178.stdout:(6/13): cockpit-podman-75-1.module+el8.9.0+2124 3.9 MB/s | 739 kB 00:00 2024-02-21T23:35:32.982 INFO:teuthology.orchestra.run.smithi178.stdout:(7/13): crun-1.8.7-1.module+el8.9.0+21243+a5865 1.6 MB/s | 239 kB 00:00 2024-02-21T23:35:33.017 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 6/13 2024-02-21T23:35:33.091 INFO:teuthology.orchestra.run.smithi178.stdout:(8/13): buildah-1.31.3-3.module+el8.9.0+21243+a 24 MB/s | 8.8 MB 00:00 2024-02-21T23:35:33.133 INFO:teuthology.orchestra.run.smithi178.stdout:(9/13): python3-podman-4.6.0-2.module+el8.9.0+2 1.0 MB/s | 154 kB 00:00 2024-02-21T23:35:33.252 INFO:teuthology.orchestra.run.smithi178.stdout:(10/13): netavark-1.7.0-2.module+el8.9.0+21243+ 13 MB/s | 3.7 MB 00:00 2024-02-21T23:35:33.278 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-podman-4.6.0-2.module+el8.9.0+21243+a58653 7/13 2024-02-21T23:35:33.333 INFO:teuthology.orchestra.run.smithi178.stdout:(11/13): toolbox-0.0.99.4-5.module+el8.9.0+2124 13 MB/s | 2.5 MB 00:00 2024-02-21T23:35:33.392 INFO:teuthology.orchestra.run.smithi178.stdout:(12/13): udica-0.2.6-20.module+el8.9.0+21243+a5 355 kB/s | 49 kB 00:00 2024-02-21T23:35:33.484 INFO:teuthology.orchestra.run.smithi178.stdout:(13/13): skopeo-1.13.3-3.module+el8.9.0+21243+a 21 MB/s | 8.1 MB 00:00 2024-02-21T23:35:33.484 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-02-21T23:35:33.485 INFO:teuthology.orchestra.run.smithi178.stdout:Total 24 MB/s | 26 MB 00:01 2024-02-21T23:35:33.771 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-02-21T23:35:33.792 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-02-21T23:35:33.792 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-02-21T23:35:33.935 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-02-21T23:35:34.020 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-02-21T23:35:34.029 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-02-21T23:35:34.093 INFO:teuthology.orchestra.run.smithi176.stdout:========================================================================================================== 2024-02-21T23:35:34.093 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-02-21T23:35:34.093 INFO:teuthology.orchestra.run.smithi176.stdout:========================================================================================================== 2024-02-21T23:35:34.093 INFO:teuthology.orchestra.run.smithi176.stdout:Installing group/module packages: 2024-02-21T23:35:34.093 INFO:teuthology.orchestra.run.smithi176.stdout: aardvark-dns x86_64 2:1.7.0-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 1.0 M 2024-02-21T23:35:34.093 INFO:teuthology.orchestra.run.smithi176.stdout: buildah x86_64 1:1.31.3-3.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 8.8 M 2024-02-21T23:35:34.093 INFO:teuthology.orchestra.run.smithi176.stdout: cockpit-podman noarch 75-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 739 k 2024-02-21T23:35:34.093 INFO:teuthology.orchestra.run.smithi176.stdout: crun x86_64 1.8.7-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 239 k 2024-02-21T23:35:34.093 INFO:teuthology.orchestra.run.smithi176.stdout: netavark x86_64 2:1.7.0-2.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-02-21T23:35:34.093 INFO:teuthology.orchestra.run.smithi176.stdout: python3-podman noarch 4.6.0-2.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 154 k 2024-02-21T23:35:34.093 INFO:teuthology.orchestra.run.smithi176.stdout: skopeo x86_64 2:1.13.3-3.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 8.1 M 2024-02-21T23:35:34.093 INFO:teuthology.orchestra.run.smithi176.stdout: toolbox x86_64 0.0.99.4-5.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 2.5 M 2024-02-21T23:35:34.093 INFO:teuthology.orchestra.run.smithi176.stdout: udica noarch 0.2.6-20.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 49 k 2024-02-21T23:35:34.094 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-02-21T23:35:34.094 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2024-02-21T23:35:34.094 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2024-02-21T23:35:34.094 INFO:teuthology.orchestra.run.smithi176.stdout: yajl x86_64 2.1.0-12.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2024-02-21T23:35:34.094 INFO:teuthology.orchestra.run.smithi176.stdout:Installing weak dependencies: 2024-02-21T23:35:34.094 INFO:teuthology.orchestra.run.smithi176.stdout: criu-libs x86_64 3.18-4.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 39 k 2024-02-21T23:35:34.094 INFO:teuthology.orchestra.run.smithi176.stdout:Installing module profiles: 2024-02-21T23:35:34.094 INFO:teuthology.orchestra.run.smithi176.stdout: container-tools/common 2024-02-21T23:35:34.094 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-21T23:35:34.094 INFO:teuthology.orchestra.run.smithi176.stdout:Enabling module streams: 2024-02-21T23:35:34.094 INFO:teuthology.orchestra.run.smithi176.stdout: container-tools rhel8 2024-02-21T23:35:34.094 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-21T23:35:34.094 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-02-21T23:35:34.095 INFO:teuthology.orchestra.run.smithi176.stdout:========================================================================================================== 2024-02-21T23:35:34.095 INFO:teuthology.orchestra.run.smithi176.stdout:Install 13 Packages 2024-02-21T23:35:34.095 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-21T23:35:34.095 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 26 M 2024-02-21T23:35:34.095 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 87 M 2024-02-21T23:35:34.095 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-02-21T23:35:34.148 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 8/13 2024-02-21T23:35:34.288 INFO:teuthology.orchestra.run.smithi176.stdout:(1/13): python3-pytoml-0.1.14-5.git7dea353.el8. 129 kB/s | 25 kB 00:00 2024-02-21T23:35:34.320 INFO:teuthology.orchestra.run.smithi176.stdout:(2/13): criu-libs-3.18-4.module+el8.9.0+20326+3 175 kB/s | 39 kB 00:00 2024-02-21T23:35:34.346 INFO:teuthology.orchestra.run.smithi176.stdout:(3/13): python3-pyxdg-0.25-16.el8.noarch.rpm 375 kB/s | 94 kB 00:00 2024-02-21T23:35:34.405 INFO:teuthology.orchestra.run.smithi176.stdout:(4/13): yajl-2.1.0-12.el8.x86_64.rpm 354 kB/s | 41 kB 00:00 2024-02-21T23:35:34.409 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x 9/13 2024-02-21T23:35:34.480 INFO:teuthology.orchestra.run.smithi176.stdout:(5/13): aardvark-dns-1.7.0-1.module+el8.9.0+212 6.3 MB/s | 1.0 MB 00:00 2024-02-21T23:35:34.580 INFO:teuthology.orchestra.run.smithi176.stdout:(6/13): cockpit-podman-75-1.module+el8.9.0+2124 4.1 MB/s | 739 kB 00:00 2024-02-21T23:35:34.605 INFO:teuthology.orchestra.run.smithi176.stdout:(7/13): crun-1.8.7-1.module+el8.9.0+21243+a5865 1.9 MB/s | 239 kB 00:00 2024-02-21T23:35:34.746 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-02-21T23:35:34.797 INFO:teuthology.orchestra.run.smithi176.stdout:(8/13): buildah-1.31.3-3.module+el8.9.0+21243+a 20 MB/s | 8.8 MB 00:00 2024-02-21T23:35:34.803 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarc 10/13 2024-02-21T23:35:34.822 INFO:teuthology.orchestra.run.smithi176.stdout:(9/13): python3-podman-4.6.0-2.module+el8.9.0+2 708 kB/s | 154 kB 00:00 2024-02-21T23:35:34.898 INFO:teuthology.orchestra.run.smithi176.stdout:(10/13): netavark-1.7.0-2.module+el8.9.0+21243+ 12 MB/s | 3.7 MB 00:00 2024-02-21T23:35:35.015 INFO:teuthology.orchestra.run.smithi176.stdout:(11/13): toolbox-0.0.99.4-5.module+el8.9.0+2124 13 MB/s | 2.5 MB 00:00 2024-02-21T23:35:35.123 INFO:teuthology.orchestra.run.smithi176.stdout:(12/13): skopeo-1.13.3-3.module+el8.9.0+21243+a 25 MB/s | 8.1 MB 00:00 2024-02-21T23:35:35.148 INFO:teuthology.orchestra.run.smithi176.stdout:(13/13): udica-0.2.6-20.module+el8.9.0+21243+a5 197 kB/s | 49 kB 00:00 2024-02-21T23:35:35.149 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-02-21T23:35:35.149 INFO:teuthology.orchestra.run.smithi176.stdout:Total 24 MB/s | 26 MB 00:01 2024-02-21T23:35:35.240 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a58653 1/13 2024-02-21T23:35:35.420 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x 2/13 2024-02-21T23:35:35.430 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-02-21T23:35:35.451 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-02-21T23:35:35.452 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-02-21T23:35:35.622 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : yajl-2.1.0-12.el8.x86_64 3/13 2024-02-21T23:35:35.680 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-02-21T23:35:35.689 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-02-21T23:35:35.694 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x 11/13 2024-02-21T23:35:35.765 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 4/13 2024-02-21T23:35:35.923 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x8 12/13 2024-02-21T23:35:35.944 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 5/13 2024-02-21T23:35:36.144 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 6/13 2024-02-21T23:35:36.156 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 13/13 2024-02-21T23:35:36.370 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-02-21T23:35:36.397 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-podman-4.6.0-2.module+el8.9.0+21243+a58653 7/13 2024-02-21T23:35:36.563 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 13/13 2024-02-21T23:35:36.563 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/13 2024-02-21T23:35:36.563 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-02-21T23:35:36.564 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 3/13 2024-02-21T23:35:36.564 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 4/13 2024-02-21T23:35:36.564 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a58653 5/13 2024-02-21T23:35:36.564 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x 6/13 2024-02-21T23:35:36.564 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 7/13 2024-02-21T23:35:36.564 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 8/13 2024-02-21T23:35:36.564 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x 9/13 2024-02-21T23:35:36.564 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-podman-4.6.0-2.module+el8.9.0+21243+a58653 10/13 2024-02-21T23:35:36.564 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x8 11/13 2024-02-21T23:35:36.564 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x 12/13 2024-02-21T23:35:36.861 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a58653 1/13 2024-02-21T23:35:37.022 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x 2/13 2024-02-21T23:35:37.207 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : yajl-2.1.0-12.el8.x86_64 3/13 2024-02-21T23:35:37.306 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 8/13 2024-02-21T23:35:37.367 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 4/13 2024-02-21T23:35:37.521 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 5/13 2024-02-21T23:35:37.551 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x 9/13 2024-02-21T23:35:37.714 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 6/13 2024-02-21T23:35:37.947 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarc 10/13 2024-02-21T23:35:37.949 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-podman-4.6.0-2.module+el8.9.0+21243+a58653 7/13 2024-02-21T23:35:38.817 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 8/13 2024-02-21T23:35:38.839 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x 11/13 2024-02-21T23:35:39.034 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x8 12/13 2024-02-21T23:35:39.044 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x 9/13 2024-02-21T23:35:39.217 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 13/13 2024-02-21T23:35:39.417 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarc 10/13 2024-02-21T23:35:39.631 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 13/13 2024-02-21T23:35:39.631 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/13 2024-02-21T23:35:39.631 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-02-21T23:35:39.631 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 3/13 2024-02-21T23:35:39.631 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 4/13 2024-02-21T23:35:39.631 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a58653 5/13 2024-02-21T23:35:39.631 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x 6/13 2024-02-21T23:35:39.631 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 7/13 2024-02-21T23:35:39.631 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 8/13 2024-02-21T23:35:39.632 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x 9/13 2024-02-21T23:35:39.632 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-podman-4.6.0-2.module+el8.9.0+21243+a58653 10/13 2024-02-21T23:35:39.632 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x8 11/13 2024-02-21T23:35:39.632 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x 12/13 2024-02-21T23:35:40.323 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x 11/13 2024-02-21T23:35:40.545 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x8 12/13 2024-02-21T23:35:40.761 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 13/13 2024-02-21T23:35:41.193 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 13/13 2024-02-21T23:35:41.193 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/13 2024-02-21T23:35:41.193 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-02-21T23:35:41.193 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 3/13 2024-02-21T23:35:41.193 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 4/13 2024-02-21T23:35:41.193 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a58653 5/13 2024-02-21T23:35:41.193 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x 6/13 2024-02-21T23:35:41.193 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 7/13 2024-02-21T23:35:41.193 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 8/13 2024-02-21T23:35:41.194 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x 9/13 2024-02-21T23:35:41.194 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-podman-4.6.0-2.module+el8.9.0+21243+a58653 10/13 2024-02-21T23:35:41.194 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x8 11/13 2024-02-21T23:35:41.194 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x 12/13 2024-02-21T23:35:47.982 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarc 13/13 2024-02-21T23:35:47.982 INFO:teuthology.orchestra.run.smithi090.stdout:Installed products updated. 2024-02-21T23:35:48.526 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:35:48.526 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-21T23:35:48.527 INFO:teuthology.orchestra.run.smithi090.stdout: aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a586538b.x86_64 2024-02-21T23:35:48.527 INFO:teuthology.orchestra.run.smithi090.stdout: buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x86_64 2024-02-21T23:35:48.527 INFO:teuthology.orchestra.run.smithi090.stdout: cockpit-podman-75-1.module+el8.9.0+21243+a586538b.noarch 2024-02-21T23:35:48.527 INFO:teuthology.orchestra.run.smithi090.stdout: criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86_64 2024-02-21T23:35:48.527 INFO:teuthology.orchestra.run.smithi090.stdout: crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 2024-02-21T23:35:48.527 INFO:teuthology.orchestra.run.smithi090.stdout: netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x86_64 2024-02-21T23:35:48.527 INFO:teuthology.orchestra.run.smithi090.stdout: python3-podman-4.6.0-2.module+el8.9.0+21243+a586538b.noarch 2024-02-21T23:35:48.527 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-21T23:35:48.527 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-21T23:35:48.527 INFO:teuthology.orchestra.run.smithi090.stdout: skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x86_64 2024-02-21T23:35:48.527 INFO:teuthology.orchestra.run.smithi090.stdout: toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x86_64 2024-02-21T23:35:48.527 INFO:teuthology.orchestra.run.smithi090.stdout: udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarch 2024-02-21T23:35:48.527 INFO:teuthology.orchestra.run.smithi090.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-21T23:35:48.527 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:35:48.527 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-21T23:35:48.666 DEBUG:teuthology.parallel:result is None 2024-02-21T23:35:50.494 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarc 13/13 2024-02-21T23:35:50.494 INFO:teuthology.orchestra.run.smithi178.stdout:Installed products updated. 2024-02-21T23:35:51.074 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-21T23:35:51.074 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-02-21T23:35:51.074 INFO:teuthology.orchestra.run.smithi178.stdout: aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a586538b.x86_64 2024-02-21T23:35:51.074 INFO:teuthology.orchestra.run.smithi178.stdout: buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x86_64 2024-02-21T23:35:51.074 INFO:teuthology.orchestra.run.smithi178.stdout: cockpit-podman-75-1.module+el8.9.0+21243+a586538b.noarch 2024-02-21T23:35:51.074 INFO:teuthology.orchestra.run.smithi178.stdout: criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86_64 2024-02-21T23:35:51.074 INFO:teuthology.orchestra.run.smithi178.stdout: crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 2024-02-21T23:35:51.074 INFO:teuthology.orchestra.run.smithi178.stdout: netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x86_64 2024-02-21T23:35:51.074 INFO:teuthology.orchestra.run.smithi178.stdout: python3-podman-4.6.0-2.module+el8.9.0+21243+a586538b.noarch 2024-02-21T23:35:51.074 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-21T23:35:51.075 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-21T23:35:51.075 INFO:teuthology.orchestra.run.smithi178.stdout: skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x86_64 2024-02-21T23:35:51.075 INFO:teuthology.orchestra.run.smithi178.stdout: toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x86_64 2024-02-21T23:35:51.075 INFO:teuthology.orchestra.run.smithi178.stdout: udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarch 2024-02-21T23:35:51.075 INFO:teuthology.orchestra.run.smithi178.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-21T23:35:51.075 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-21T23:35:51.075 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-02-21T23:35:51.219 DEBUG:teuthology.parallel:result is None 2024-02-21T23:35:53.776 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarc 13/13 2024-02-21T23:35:53.776 INFO:teuthology.orchestra.run.smithi176.stdout:Installed products updated. 2024-02-21T23:35:54.323 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-21T23:35:54.323 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-02-21T23:35:54.324 INFO:teuthology.orchestra.run.smithi176.stdout: aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a586538b.x86_64 2024-02-21T23:35:54.324 INFO:teuthology.orchestra.run.smithi176.stdout: buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x86_64 2024-02-21T23:35:54.324 INFO:teuthology.orchestra.run.smithi176.stdout: cockpit-podman-75-1.module+el8.9.0+21243+a586538b.noarch 2024-02-21T23:35:54.324 INFO:teuthology.orchestra.run.smithi176.stdout: criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86_64 2024-02-21T23:35:54.324 INFO:teuthology.orchestra.run.smithi176.stdout: crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 2024-02-21T23:35:54.324 INFO:teuthology.orchestra.run.smithi176.stdout: netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x86_64 2024-02-21T23:35:54.324 INFO:teuthology.orchestra.run.smithi176.stdout: python3-podman-4.6.0-2.module+el8.9.0+21243+a586538b.noarch 2024-02-21T23:35:54.324 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-21T23:35:54.325 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-21T23:35:54.325 INFO:teuthology.orchestra.run.smithi176.stdout: skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x86_64 2024-02-21T23:35:54.325 INFO:teuthology.orchestra.run.smithi176.stdout: toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x86_64 2024-02-21T23:35:54.325 INFO:teuthology.orchestra.run.smithi176.stdout: udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarch 2024-02-21T23:35:54.325 INFO:teuthology.orchestra.run.smithi176.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-21T23:35:54.325 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-21T23:35:54.325 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-02-21T23:35:54.473 DEBUG:teuthology.parallel:result is None 2024-02-21T23:35:54.473 INFO:teuthology.run_tasks:Running task install... 2024-02-21T23:35:54.483 DEBUG:teuthology.task.install:project ceph 2024-02-21T23:35:54.483 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'c1dc18f444603397768fcb7c34a8488b10284df6'}} 2024-02-21T23:35:54.483 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'c1dc18f444603397768fcb7c34a8488b10284df6'} 2024-02-21T23:35:54.483 INFO:teuthology.task.install:Using flavor: default 2024-02-21T23:35:54.495 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-21T23:35:54.495 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2024-02-21T23:35:54.496 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c1dc18f444603397768fcb7c34a8488b10284df6', 'tag': None, 'wait_for_package': False} 2024-02-21T23:35:54.496 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=c1dc18f444603397768fcb7c34a8488b10284df6 2024-02-21T23:35:54.499 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c1dc18f444603397768fcb7c34a8488b10284df6', 'tag': None, 'wait_for_package': False} 2024-02-21T23:35:54.499 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=c1dc18f444603397768fcb7c34a8488b10284df6 2024-02-21T23:35:54.500 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c1dc18f444603397768fcb7c34a8488b10284df6', 'tag': None, 'wait_for_package': False} 2024-02-21T23:35:54.500 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=c1dc18f444603397768fcb7c34a8488b10284df6 2024-02-21T23:35:54.652 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-02-21-1249-reef/c1dc18f444603397768fcb7c34a8488b10284df6/centos/8/flavors/default/ 2024-02-21T23:35:54.652 INFO:teuthology.task.install.rpm:Package version is 18.2.1-638.gc1dc18f4 2024-02-21T23:35:54.719 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-02-21-1249-reef/c1dc18f444603397768fcb7c34a8488b10284df6/centos/8/flavors/default/ 2024-02-21T23:35:54.719 INFO:teuthology.task.install.rpm:Package version is 18.2.1-638.gc1dc18f4 2024-02-21T23:35:54.724 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-02-21-1249-reef/c1dc18f444603397768fcb7c34a8488b10284df6/centos/8/flavors/default/ 2024-02-21T23:35:54.724 INFO:teuthology.task.install.rpm:Package version is 18.2.1-638.gc1dc18f4 2024-02-21T23:35:54.827 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-02-21-1249-reef/c1dc18f444603397768fcb7c34a8488b10284df6/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-02-21-1249-reef/c1dc18f444603397768fcb7c34a8488b10284df6/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-02-21-1249-reef/c1dc18f444603397768fcb7c34a8488b10284df6/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-21T23:35:54.827 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-21T23:35:54.827 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-21T23:35:54.832 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-02-21-1249-reef/c1dc18f444603397768fcb7c34a8488b10284df6/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-02-21-1249-reef/c1dc18f444603397768fcb7c34a8488b10284df6/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-02-21-1249-reef/c1dc18f444603397768fcb7c34a8488b10284df6/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-21T23:35:54.832 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-02-21T23:35:54.832 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-21T23:35:54.860 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-02-21T23:35:54.861 DEBUG:teuthology.orchestra.run.smithi090:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c1dc18f444603397768fcb7c34a8488b10284df6/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-21T23:35:54.865 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-02-21T23:35:54.865 DEBUG:teuthology.orchestra.run.smithi178:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c1dc18f444603397768fcb7c34a8488b10284df6/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-21T23:35:54.892 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-02-21-1249-reef/c1dc18f444603397768fcb7c34a8488b10284df6/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-02-21-1249-reef/c1dc18f444603397768fcb7c34a8488b10284df6/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-02-21-1249-reef/c1dc18f444603397768fcb7c34a8488b10284df6/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-21T23:35:54.893 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-02-21T23:35:54.893 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-21T23:35:54.921 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-02-21T23:35:54.921 DEBUG:teuthology.orchestra.run.smithi176:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c1dc18f444603397768fcb7c34a8488b10284df6/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-21T23:35:54.931 DEBUG:teuthology.orchestra.run.smithi090:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-21T23:35:54.939 DEBUG:teuthology.orchestra.run.smithi178:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-21T23:35:54.989 DEBUG:teuthology.orchestra.run.smithi176:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-21T23:35:55.012 DEBUG:teuthology.orchestra.run.smithi090:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-21T23:35:55.021 DEBUG:teuthology.orchestra.run.smithi178:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-21T23:35:55.067 DEBUG:teuthology.orchestra.run.smithi176:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-21T23:35:55.080 INFO:teuthology.orchestra.run.smithi090.stdout:check_obsoletes = 1 2024-02-21T23:35:55.081 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum clean all 2024-02-21T23:35:55.090 INFO:teuthology.orchestra.run.smithi178.stdout:check_obsoletes = 1 2024-02-21T23:35:55.091 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum clean all 2024-02-21T23:35:55.133 INFO:teuthology.orchestra.run.smithi176.stdout:check_obsoletes = 1 2024-02-21T23:35:55.135 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum clean all 2024-02-21T23:35:55.460 INFO:teuthology.orchestra.run.smithi090.stdout:Updating Subscription Management repositories. 2024-02-21T23:35:55.471 INFO:teuthology.orchestra.run.smithi178.stdout:Updating Subscription Management repositories. 2024-02-21T23:35:55.513 INFO:teuthology.orchestra.run.smithi176.stdout:Updating Subscription Management repositories. 2024-02-21T23:35:56.772 INFO:teuthology.orchestra.run.smithi178.stdout:61 files removed 2024-02-21T23:35:56.791 INFO:teuthology.orchestra.run.smithi176.stdout:61 files removed 2024-02-21T23:35:56.824 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-02-21T23:35:56.842 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-02-21T23:35:56.872 INFO:teuthology.orchestra.run.smithi090.stdout:61 files removed 2024-02-21T23:35:56.925 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-02-21T23:35:57.162 INFO:teuthology.orchestra.run.smithi178.stdout:Updating Subscription Management repositories. 2024-02-21T23:35:57.179 INFO:teuthology.orchestra.run.smithi176.stdout:Updating Subscription Management repositories. 2024-02-21T23:35:57.262 INFO:teuthology.orchestra.run.smithi090.stdout:Updating Subscription Management repositories. 2024-02-21T23:35:58.916 INFO:teuthology.orchestra.run.smithi178.stdout:ceph packages for x86_64 200 kB/s | 80 kB 00:00 2024-02-21T23:35:59.003 INFO:teuthology.orchestra.run.smithi176.stdout:ceph packages for x86_64 224 kB/s | 80 kB 00:00 2024-02-21T23:35:59.060 INFO:teuthology.orchestra.run.smithi090.stdout:ceph packages for x86_64 255 kB/s | 80 kB 00:00 2024-02-21T23:35:59.225 INFO:teuthology.orchestra.run.smithi178.stdout:ceph noarch packages 36 kB/s | 9.9 kB 00:00 2024-02-21T23:35:59.337 INFO:teuthology.orchestra.run.smithi176.stdout:ceph noarch packages 33 kB/s | 9.9 kB 00:00 2024-02-21T23:35:59.378 INFO:teuthology.orchestra.run.smithi090.stdout:ceph noarch packages 35 kB/s | 9.9 kB 00:00 2024-02-21T23:35:59.492 INFO:teuthology.orchestra.run.smithi178.stdout:ceph source packages 6.2 kB/s | 1.5 kB 00:00 2024-02-21T23:35:59.694 INFO:teuthology.orchestra.run.smithi178.stdout:Copr repo for python3-asyncssh owned by ceph 20 kB/s | 3.5 kB 00:00 2024-02-21T23:35:59.712 INFO:teuthology.orchestra.run.smithi176.stdout:ceph source packages 4.3 kB/s | 1.5 kB 00:00 2024-02-21T23:35:59.754 INFO:teuthology.orchestra.run.smithi090.stdout:ceph source packages 4.3 kB/s | 1.5 kB 00:00 2024-02-21T23:35:59.863 INFO:teuthology.orchestra.run.smithi176.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2024-02-21T23:35:59.896 INFO:teuthology.orchestra.run.smithi090.stdout:Copr repo for python3-asyncssh owned by ceph 29 kB/s | 3.5 kB 00:00 2024-02-21T23:36:00.872 INFO:teuthology.orchestra.run.smithi176.stdout:Extra Packages for Enterprise Linux 16 MB/s | 16 MB 00:00 2024-02-21T23:36:02.860 INFO:teuthology.orchestra.run.smithi178.stdout:Extra Packages for Enterprise Linux 5.1 MB/s | 16 MB 00:03 2024-02-21T23:36:04.489 INFO:teuthology.orchestra.run.smithi176.stdout:lab-extras 294 kB/s | 24 kB 00:00 2024-02-21T23:36:05.724 INFO:teuthology.orchestra.run.smithi176.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 51 MB/s | 59 MB 00:01 2024-02-21T23:36:06.385 INFO:teuthology.orchestra.run.smithi178.stdout:lab-extras 400 kB/s | 24 kB 00:00 2024-02-21T23:36:07.979 INFO:teuthology.orchestra.run.smithi178.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 39 MB/s | 59 MB 00:01 2024-02-21T23:36:16.208 INFO:teuthology.orchestra.run.smithi176.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 14 MB/s | 8.8 MB 00:00 2024-02-21T23:36:18.321 INFO:teuthology.orchestra.run.smithi178.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 17 MB/s | 8.8 MB 00:00 2024-02-21T23:36:19.035 INFO:teuthology.orchestra.run.smithi176.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 54 MB/s | 65 MB 00:01 2024-02-21T23:36:21.539 INFO:teuthology.orchestra.run.smithi178.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 43 MB/s | 65 MB 00:01 2024-02-21T23:36:34.120 INFO:teuthology.orchestra.run.smithi176.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-02-21T23:36:34.120 INFO:teuthology.orchestra.run.smithi176.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-02-21T23:36:34.139 INFO:teuthology.orchestra.run.smithi176.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2024-02-21T23:36:34.140 INFO:teuthology.orchestra.run.smithi176.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2024-02-21T23:36:34.141 INFO:teuthology.orchestra.run.smithi176.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2024-02-21T23:36:34.143 INFO:teuthology.orchestra.run.smithi176.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-02-21T23:36:34.145 INFO:teuthology.orchestra.run.smithi176.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-02-21T23:36:34.149 INFO:teuthology.orchestra.run.smithi176.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-02-21T23:36:34.150 INFO:teuthology.orchestra.run.smithi176.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-02-21T23:36:34.359 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-02-21T23:36:34.373 INFO:teuthology.orchestra.run.smithi176.stdout:=================================================================================================================================== 2024-02-21T23:36:34.373 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-02-21T23:36:34.373 INFO:teuthology.orchestra.run.smithi176.stdout:=================================================================================================================================== 2024-02-21T23:36:34.373 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-02-21T23:36:34.373 INFO:teuthology.orchestra.run.smithi176.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-13.el8_9 rhel-8-for-x86_64-baseos-rpms 144 k 2024-02-21T23:36:34.373 INFO:teuthology.orchestra.run.smithi176.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2024-02-21T23:36:34.374 INFO:teuthology.orchestra.run.smithi176.stdout: ceph x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 6.4 k 2024-02-21T23:36:34.374 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-base x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 5.4 M 2024-02-21T23:36:34.374 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-fuse x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 853 k 2024-02-21T23:36:34.374 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-immutable-object-cache x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 165 k 2024-02-21T23:36:34.374 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 1.6 M 2024-02-21T23:36:34.374 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-cephadm noarch 2:18.2.1-638.gc1dc18f4.el8 ceph-noarch 137 k 2024-02-21T23:36:34.374 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-dashboard noarch 2:18.2.1-638.gc1dc18f4.el8 ceph-noarch 1.7 M 2024-02-21T23:36:34.374 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.1-638.gc1dc18f4.el8 ceph-noarch 7.4 M 2024-02-21T23:36:34.374 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-rook noarch 2:18.2.1-638.gc1dc18f4.el8 ceph-noarch 52 k 2024-02-21T23:36:34.374 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-radosgw x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 13 M 2024-02-21T23:36:34.374 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-test x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 47 M 2024-02-21T23:36:34.374 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-volume noarch 2:18.2.1-638.gc1dc18f4.el8 ceph-noarch 263 k 2024-02-21T23:36:34.374 INFO:teuthology.orchestra.run.smithi176.stdout: cephadm noarch 2:18.2.1-638.gc1dc18f4.el8 ceph-noarch 172 k 2024-02-21T23:36:34.374 INFO:teuthology.orchestra.run.smithi176.stdout: cephfs-mirror x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 259 k 2024-02-21T23:36:34.375 INFO:teuthology.orchestra.run.smithi176.stdout: cephfs-top noarch 2:18.2.1-638.gc1dc18f4.el8 ceph-noarch 18 k 2024-02-21T23:36:34.375 INFO:teuthology.orchestra.run.smithi176.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2024-02-21T23:36:34.375 INFO:teuthology.orchestra.run.smithi176.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2024-02-21T23:36:34.375 INFO:teuthology.orchestra.run.smithi176.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2024-02-21T23:36:34.375 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs-devel x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 31 k 2024-02-21T23:36:34.375 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs2 x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 775 k 2024-02-21T23:36:34.375 INFO:teuthology.orchestra.run.smithi176.stdout: librados-devel x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 139 k 2024-02-21T23:36:34.375 INFO:teuthology.orchestra.run.smithi176.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2024-02-21T23:36:34.375 INFO:teuthology.orchestra.run.smithi176.stdout: libuuid-devel x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 100 k 2024-02-21T23:36:34.375 INFO:teuthology.orchestra.run.smithi176.stdout: openssl-devel x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 2.3 M 2024-02-21T23:36:34.375 INFO:teuthology.orchestra.run.smithi176.stdout: postgresql x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 1.5 M 2024-02-21T23:36:34.375 INFO:teuthology.orchestra.run.smithi176.stdout: postgresql-contrib x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 811 k 2024-02-21T23:36:34.376 INFO:teuthology.orchestra.run.smithi176.stdout: postgresql-server x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 5.1 M 2024-02-21T23:36:34.376 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cephfs x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 196 k 2024-02-21T23:36:34.376 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rados x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 371 k 2024-02-21T23:36:34.376 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rbd x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 364 k 2024-02-21T23:36:34.376 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rgw x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 106 k 2024-02-21T23:36:34.376 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-fuse x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 95 k 2024-02-21T23:36:34.376 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-mirror x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 3.5 M 2024-02-21T23:36:34.376 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-nbd x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 182 k 2024-02-21T23:36:34.376 INFO:teuthology.orchestra.run.smithi176.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 73 k 2024-02-21T23:36:34.376 INFO:teuthology.orchestra.run.smithi176.stdout:Upgrading: 2024-02-21T23:36:34.376 INFO:teuthology.orchestra.run.smithi176.stdout: elfutils-libelf x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 232 k 2024-02-21T23:36:34.376 INFO:teuthology.orchestra.run.smithi176.stdout: elfutils-libs x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 303 k 2024-02-21T23:36:34.376 INFO:teuthology.orchestra.run.smithi176.stdout: iproute x86_64 6.2.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 854 k 2024-02-21T23:36:34.377 INFO:teuthology.orchestra.run.smithi176.stdout: libblkid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-02-21T23:36:34.377 INFO:teuthology.orchestra.run.smithi176.stdout: libfdisk x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 254 k 2024-02-21T23:36:34.377 INFO:teuthology.orchestra.run.smithi176.stdout: libmount x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 237 k 2024-02-21T23:36:34.377 INFO:teuthology.orchestra.run.smithi176.stdout: librados2 x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 3.7 M 2024-02-21T23:36:34.377 INFO:teuthology.orchestra.run.smithi176.stdout: librbd1 x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 3.6 M 2024-02-21T23:36:34.377 INFO:teuthology.orchestra.run.smithi176.stdout: libsmartcols x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 179 k 2024-02-21T23:36:34.377 INFO:teuthology.orchestra.run.smithi176.stdout: libuuid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 99 k 2024-02-21T23:36:34.377 INFO:teuthology.orchestra.run.smithi176.stdout: openssl x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 711 k 2024-02-21T23:36:34.377 INFO:teuthology.orchestra.run.smithi176.stdout: openssl-libs x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 1.5 M 2024-02-21T23:36:34.377 INFO:teuthology.orchestra.run.smithi176.stdout: util-linux x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 2.5 M 2024-02-21T23:36:34.377 INFO:teuthology.orchestra.run.smithi176.stdout: util-linux-user x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 103 k 2024-02-21T23:36:34.377 INFO:teuthology.orchestra.run.smithi176.stdout: xfsdump x86_64 3.1.8-7.el8_9 rhel-8-for-x86_64-baseos-rpms 333 k 2024-02-21T23:36:34.377 INFO:teuthology.orchestra.run.smithi176.stdout: xfsprogs x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M 2024-02-21T23:36:34.378 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-02-21T23:36:34.378 INFO:teuthology.orchestra.run.smithi176.stdout: automake noarch 1.16.1-8.el8 rhel-8-for-x86_64-appstream-rpms 714 k 2024-02-21T23:36:34.378 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-common x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 22 M 2024-02-21T23:36:34.378 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-grafana-dashboards noarch 2:18.2.1-638.gc1dc18f4.el8 ceph-noarch 23 k 2024-02-21T23:36:34.378 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mds x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 2.3 M 2024-02-21T23:36:34.378 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-modules-core noarch 2:18.2.1-638.gc1dc18f4.el8 ceph-noarch 244 k 2024-02-21T23:36:34.378 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mon x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 4.6 M 2024-02-21T23:36:34.378 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-osd x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 18 M 2024-02-21T23:36:34.378 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-prometheus-alerts noarch 2:18.2.1-638.gc1dc18f4.el8 ceph-noarch 15 k 2024-02-21T23:36:34.378 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-selinux x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 24 k 2024-02-21T23:36:34.378 INFO:teuthology.orchestra.run.smithi176.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-02-21T23:36:34.378 INFO:teuthology.orchestra.run.smithi176.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-21T23:36:34.378 INFO:teuthology.orchestra.run.smithi176.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-02-21T23:36:34.379 INFO:teuthology.orchestra.run.smithi176.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2024-02-21T23:36:34.379 INFO:teuthology.orchestra.run.smithi176.stdout: krb5-devel x86_64 1.18.2-26.el8_9 rhel-8-for-x86_64-baseos-rpms 562 k 2024-02-21T23:36:34.379 INFO:teuthology.orchestra.run.smithi176.stdout: libcephsqlite x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 142 k 2024-02-21T23:36:34.379 INFO:teuthology.orchestra.run.smithi176.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2024-02-21T23:36:34.379 INFO:teuthology.orchestra.run.smithi176.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-02-21T23:36:34.379 INFO:teuthology.orchestra.run.smithi176.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-21T23:36:34.379 INFO:teuthology.orchestra.run.smithi176.stdout: libpq x86_64 13.11-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2024-02-21T23:36:34.379 INFO:teuthology.orchestra.run.smithi176.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-02-21T23:36:34.379 INFO:teuthology.orchestra.run.smithi176.stdout: librabbitmq x86_64 0.9.0-4.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2024-02-21T23:36:34.379 INFO:teuthology.orchestra.run.smithi176.stdout: libradosstriper1 x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 473 k 2024-02-21T23:36:34.379 INFO:teuthology.orchestra.run.smithi176.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-02-21T23:36:34.379 INFO:teuthology.orchestra.run.smithi176.stdout: librgw2 x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 6.5 M 2024-02-21T23:36:34.379 INFO:teuthology.orchestra.run.smithi176.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2024-02-21T23:36:34.379 INFO:teuthology.orchestra.run.smithi176.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2024-02-21T23:36:34.380 INFO:teuthology.orchestra.run.smithi176.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-21T23:36:34.380 INFO:teuthology.orchestra.run.smithi176.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2024-02-21T23:36:34.380 INFO:teuthology.orchestra.run.smithi176.stdout: libzstd-devel x86_64 1.4.4-1.el8 rhel-8-for-x86_64-baseos-rpms 44 k 2024-02-21T23:36:34.380 INFO:teuthology.orchestra.run.smithi176.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-02-21T23:36:34.380 INFO:teuthology.orchestra.run.smithi176.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2024-02-21T23:36:34.380 INFO:teuthology.orchestra.run.smithi176.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-02-21T23:36:34.380 INFO:teuthology.orchestra.run.smithi176.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-02-21T23:36:34.380 INFO:teuthology.orchestra.run.smithi176.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2024-02-21T23:36:34.380 INFO:teuthology.orchestra.run.smithi176.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2024-02-21T23:36:34.380 INFO:teuthology.orchestra.run.smithi176.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-02-21T23:36:34.380 INFO:teuthology.orchestra.run.smithi176.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-02-21T23:36:34.380 INFO:teuthology.orchestra.run.smithi176.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-02-21T23:36:34.381 INFO:teuthology.orchestra.run.smithi176.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-21T23:36:34.381 INFO:teuthology.orchestra.run.smithi176.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-21T23:36:34.381 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-21T23:36:34.381 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-argparse x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 45 k 2024-02-21T23:36:34.381 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-common x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 125 k 2024-02-21T23:36:34.381 INFO:teuthology.orchestra.run.smithi176.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-21T23:36:34.381 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-02-21T23:36:34.381 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-21T23:36:34.381 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-21T23:36:34.381 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cryptography x86_64 3.2.1-6.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2024-02-21T23:36:34.381 INFO:teuthology.orchestra.run.smithi176.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-21T23:36:34.381 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-21T23:36:34.381 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-21T23:36:34.381 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-02-21T23:36:34.382 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-02-21T23:36:34.382 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-02-21T23:36:34.382 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-02-21T23:36:34.382 INFO:teuthology.orchestra.run.smithi176.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-21T23:36:34.382 INFO:teuthology.orchestra.run.smithi176.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-21T23:36:34.382 INFO:teuthology.orchestra.run.smithi176.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-02-21T23:36:34.382 INFO:teuthology.orchestra.run.smithi176.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-02-21T23:36:34.382 INFO:teuthology.orchestra.run.smithi176.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-21T23:36:34.382 INFO:teuthology.orchestra.run.smithi176.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-02-21T23:36:34.382 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-02-21T23:36:34.382 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-02-21T23:36:34.382 INFO:teuthology.orchestra.run.smithi176.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-02-21T23:36:34.382 INFO:teuthology.orchestra.run.smithi176.stdout: python3-packaging noarch 16.8-9.el8 codeready-builder-for-rhel-8-x86_64-rpms 48 k 2024-02-21T23:36:34.382 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-21T23:36:34.383 INFO:teuthology.orchestra.run.smithi176.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-21T23:36:34.383 INFO:teuthology.orchestra.run.smithi176.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-02-21T23:36:34.383 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-02-21T23:36:34.383 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-02-21T23:36:34.383 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-02-21T23:36:34.383 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-02-21T23:36:34.383 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-02-21T23:36:34.383 INFO:teuthology.orchestra.run.smithi176.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-21T23:36:34.383 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-02-21T23:36:34.384 INFO:teuthology.orchestra.run.smithi176.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-21T23:36:34.384 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-21T23:36:34.384 INFO:teuthology.orchestra.run.smithi176.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-02-21T23:36:34.384 INFO:teuthology.orchestra.run.smithi176.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-21T23:36:34.384 INFO:teuthology.orchestra.run.smithi176.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-21T23:36:34.384 INFO:teuthology.orchestra.run.smithi176.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-21T23:36:34.384 INFO:teuthology.orchestra.run.smithi176.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-21T23:36:34.384 INFO:teuthology.orchestra.run.smithi176.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-21T23:36:34.384 INFO:teuthology.orchestra.run.smithi176.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-21T23:36:34.385 INFO:teuthology.orchestra.run.smithi176.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-21T23:36:34.385 INFO:teuthology.orchestra.run.smithi176.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-21T23:36:34.385 INFO:teuthology.orchestra.run.smithi176.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-02-21T23:36:34.385 INFO:teuthology.orchestra.run.smithi176.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-21T23:36:34.385 INFO:teuthology.orchestra.run.smithi176.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-02-21T23:36:34.385 INFO:teuthology.orchestra.run.smithi176.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-02-21T23:36:34.385 INFO:teuthology.orchestra.run.smithi176.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2024-02-21T23:36:34.385 INFO:teuthology.orchestra.run.smithi176.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-02-21T23:36:34.385 INFO:teuthology.orchestra.run.smithi176.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2024-02-21T23:36:34.385 INFO:teuthology.orchestra.run.smithi176.stdout:Enabling module streams: 2024-02-21T23:36:34.385 INFO:teuthology.orchestra.run.smithi176.stdout: postgresql 10 2024-02-21T23:36:34.386 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-21T23:36:34.386 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-02-21T23:36:34.386 INFO:teuthology.orchestra.run.smithi176.stdout:=================================================================================================================================== 2024-02-21T23:36:34.386 INFO:teuthology.orchestra.run.smithi176.stdout:Install 130 Packages 2024-02-21T23:36:34.386 INFO:teuthology.orchestra.run.smithi176.stdout:Upgrade 16 Packages 2024-02-21T23:36:34.386 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-21T23:36:34.387 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 212 M 2024-02-21T23:36:34.387 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-02-21T23:36:34.642 INFO:teuthology.orchestra.run.smithi176.stdout:(1/146): ceph-18.2.1-638.gc1dc18f4.el8.x86_64.r 36 kB/s | 6.4 kB 00:00 2024-02-21T23:36:34.951 INFO:teuthology.orchestra.run.smithi176.stdout:(2/146): ceph-base-18.2.1-638.gc1dc18f4.el8.x86 11 MB/s | 5.4 MB 00:00 2024-02-21T23:36:34.992 INFO:teuthology.orchestra.run.smithi176.stdout:(3/146): ceph-immutable-object-cache-18.2.1-638 3.9 MB/s | 165 kB 00:00 2024-02-21T23:36:35.143 INFO:teuthology.orchestra.run.smithi176.stdout:(4/146): ceph-mds-18.2.1-638.gc1dc18f4.el8.x86_ 15 MB/s | 2.3 MB 00:00 2024-02-21T23:36:35.243 INFO:teuthology.orchestra.run.smithi176.stdout:(5/146): ceph-mgr-18.2.1-638.gc1dc18f4.el8.x86_ 16 MB/s | 1.6 MB 00:00 2024-02-21T23:36:35.285 INFO:teuthology.orchestra.run.smithi176.stdout:(6/146): ceph-fuse-18.2.1-638.gc1dc18f4.el8.x86 1.3 MB/s | 853 kB 00:00 2024-02-21T23:36:35.486 INFO:teuthology.orchestra.run.smithi176.stdout:(7/146): ceph-mon-18.2.1-638.gc1dc18f4.el8.x86_ 19 MB/s | 4.6 MB 00:00 2024-02-21T23:36:36.202 INFO:teuthology.orchestra.run.smithi176.stdout:(8/146): ceph-radosgw-18.2.1-638.gc1dc18f4.el8. 18 MB/s | 13 MB 00:00 2024-02-21T23:36:36.253 INFO:teuthology.orchestra.run.smithi176.stdout:(9/146): ceph-selinux-18.2.1-638.gc1dc18f4.el8. 486 kB/s | 24 kB 00:00 2024-02-21T23:36:36.495 INFO:teuthology.orchestra.run.smithi178.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-02-21T23:36:36.496 INFO:teuthology.orchestra.run.smithi178.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-02-21T23:36:36.514 INFO:teuthology.orchestra.run.smithi178.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2024-02-21T23:36:36.515 INFO:teuthology.orchestra.run.smithi178.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2024-02-21T23:36:36.517 INFO:teuthology.orchestra.run.smithi178.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2024-02-21T23:36:36.518 INFO:teuthology.orchestra.run.smithi178.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-02-21T23:36:36.521 INFO:teuthology.orchestra.run.smithi178.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-02-21T23:36:36.525 INFO:teuthology.orchestra.run.smithi178.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-02-21T23:36:36.526 INFO:teuthology.orchestra.run.smithi178.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-02-21T23:36:36.739 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-02-21T23:36:36.754 INFO:teuthology.orchestra.run.smithi178.stdout:=================================================================================================================================== 2024-02-21T23:36:36.754 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-02-21T23:36:36.754 INFO:teuthology.orchestra.run.smithi178.stdout:=================================================================================================================================== 2024-02-21T23:36:36.754 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-02-21T23:36:36.754 INFO:teuthology.orchestra.run.smithi178.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-13.el8_9 rhel-8-for-x86_64-baseos-rpms 144 k 2024-02-21T23:36:36.755 INFO:teuthology.orchestra.run.smithi178.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2024-02-21T23:36:36.755 INFO:teuthology.orchestra.run.smithi178.stdout: ceph x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 6.4 k 2024-02-21T23:36:36.755 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-base x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 5.4 M 2024-02-21T23:36:36.755 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-fuse x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 853 k 2024-02-21T23:36:36.755 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-immutable-object-cache x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 165 k 2024-02-21T23:36:36.755 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 1.6 M 2024-02-21T23:36:36.755 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-cephadm noarch 2:18.2.1-638.gc1dc18f4.el8 ceph-noarch 137 k 2024-02-21T23:36:36.755 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-dashboard noarch 2:18.2.1-638.gc1dc18f4.el8 ceph-noarch 1.7 M 2024-02-21T23:36:36.755 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.1-638.gc1dc18f4.el8 ceph-noarch 7.4 M 2024-02-21T23:36:36.755 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-rook noarch 2:18.2.1-638.gc1dc18f4.el8 ceph-noarch 52 k 2024-02-21T23:36:36.755 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-radosgw x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 13 M 2024-02-21T23:36:36.755 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-test x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 47 M 2024-02-21T23:36:36.756 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-volume noarch 2:18.2.1-638.gc1dc18f4.el8 ceph-noarch 263 k 2024-02-21T23:36:36.756 INFO:teuthology.orchestra.run.smithi178.stdout: cephadm noarch 2:18.2.1-638.gc1dc18f4.el8 ceph-noarch 172 k 2024-02-21T23:36:36.756 INFO:teuthology.orchestra.run.smithi178.stdout: cephfs-mirror x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 259 k 2024-02-21T23:36:36.756 INFO:teuthology.orchestra.run.smithi178.stdout: cephfs-top noarch 2:18.2.1-638.gc1dc18f4.el8 ceph-noarch 18 k 2024-02-21T23:36:36.756 INFO:teuthology.orchestra.run.smithi178.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2024-02-21T23:36:36.756 INFO:teuthology.orchestra.run.smithi178.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2024-02-21T23:36:36.756 INFO:teuthology.orchestra.run.smithi178.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2024-02-21T23:36:36.756 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs-devel x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 31 k 2024-02-21T23:36:36.756 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs2 x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 775 k 2024-02-21T23:36:36.756 INFO:teuthology.orchestra.run.smithi178.stdout: librados-devel x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 139 k 2024-02-21T23:36:36.756 INFO:teuthology.orchestra.run.smithi178.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2024-02-21T23:36:36.757 INFO:teuthology.orchestra.run.smithi178.stdout: libuuid-devel x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 100 k 2024-02-21T23:36:36.757 INFO:teuthology.orchestra.run.smithi178.stdout: openssl-devel x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 2.3 M 2024-02-21T23:36:36.757 INFO:teuthology.orchestra.run.smithi178.stdout: postgresql x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 1.5 M 2024-02-21T23:36:36.757 INFO:teuthology.orchestra.run.smithi178.stdout: postgresql-contrib x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 811 k 2024-02-21T23:36:36.757 INFO:teuthology.orchestra.run.smithi178.stdout: postgresql-server x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 5.1 M 2024-02-21T23:36:36.757 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cephfs x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 196 k 2024-02-21T23:36:36.757 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rados x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 371 k 2024-02-21T23:36:36.757 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rbd x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 364 k 2024-02-21T23:36:36.757 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rgw x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 106 k 2024-02-21T23:36:36.757 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-fuse x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 95 k 2024-02-21T23:36:36.757 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-mirror x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 3.5 M 2024-02-21T23:36:36.757 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-nbd x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 182 k 2024-02-21T23:36:36.757 INFO:teuthology.orchestra.run.smithi178.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 73 k 2024-02-21T23:36:36.758 INFO:teuthology.orchestra.run.smithi178.stdout:Upgrading: 2024-02-21T23:36:36.758 INFO:teuthology.orchestra.run.smithi178.stdout: elfutils-libelf x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 232 k 2024-02-21T23:36:36.758 INFO:teuthology.orchestra.run.smithi178.stdout: elfutils-libs x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 303 k 2024-02-21T23:36:36.758 INFO:teuthology.orchestra.run.smithi178.stdout: iproute x86_64 6.2.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 854 k 2024-02-21T23:36:36.758 INFO:teuthology.orchestra.run.smithi178.stdout: libblkid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-02-21T23:36:36.758 INFO:teuthology.orchestra.run.smithi178.stdout: libfdisk x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 254 k 2024-02-21T23:36:36.758 INFO:teuthology.orchestra.run.smithi178.stdout: libmount x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 237 k 2024-02-21T23:36:36.758 INFO:teuthology.orchestra.run.smithi178.stdout: librados2 x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 3.7 M 2024-02-21T23:36:36.758 INFO:teuthology.orchestra.run.smithi178.stdout: librbd1 x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 3.6 M 2024-02-21T23:36:36.758 INFO:teuthology.orchestra.run.smithi178.stdout: libsmartcols x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 179 k 2024-02-21T23:36:36.758 INFO:teuthology.orchestra.run.smithi178.stdout: libuuid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 99 k 2024-02-21T23:36:36.758 INFO:teuthology.orchestra.run.smithi178.stdout: openssl x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 711 k 2024-02-21T23:36:36.758 INFO:teuthology.orchestra.run.smithi178.stdout: openssl-libs x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 1.5 M 2024-02-21T23:36:36.758 INFO:teuthology.orchestra.run.smithi178.stdout: util-linux x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 2.5 M 2024-02-21T23:36:36.759 INFO:teuthology.orchestra.run.smithi178.stdout: util-linux-user x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 103 k 2024-02-21T23:36:36.759 INFO:teuthology.orchestra.run.smithi178.stdout: xfsdump x86_64 3.1.8-7.el8_9 rhel-8-for-x86_64-baseos-rpms 333 k 2024-02-21T23:36:36.759 INFO:teuthology.orchestra.run.smithi178.stdout: xfsprogs x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M 2024-02-21T23:36:36.759 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-02-21T23:36:36.759 INFO:teuthology.orchestra.run.smithi178.stdout: automake noarch 1.16.1-8.el8 rhel-8-for-x86_64-appstream-rpms 714 k 2024-02-21T23:36:36.759 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-common x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 22 M 2024-02-21T23:36:36.759 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-grafana-dashboards noarch 2:18.2.1-638.gc1dc18f4.el8 ceph-noarch 23 k 2024-02-21T23:36:36.759 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mds x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 2.3 M 2024-02-21T23:36:36.759 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-modules-core noarch 2:18.2.1-638.gc1dc18f4.el8 ceph-noarch 244 k 2024-02-21T23:36:36.759 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mon x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 4.6 M 2024-02-21T23:36:36.759 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-osd x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 18 M 2024-02-21T23:36:36.759 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-prometheus-alerts noarch 2:18.2.1-638.gc1dc18f4.el8 ceph-noarch 15 k 2024-02-21T23:36:36.759 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-selinux x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 24 k 2024-02-21T23:36:36.760 INFO:teuthology.orchestra.run.smithi178.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-02-21T23:36:36.760 INFO:teuthology.orchestra.run.smithi178.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-21T23:36:36.760 INFO:teuthology.orchestra.run.smithi178.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-02-21T23:36:36.760 INFO:teuthology.orchestra.run.smithi178.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2024-02-21T23:36:36.760 INFO:teuthology.orchestra.run.smithi178.stdout: krb5-devel x86_64 1.18.2-26.el8_9 rhel-8-for-x86_64-baseos-rpms 562 k 2024-02-21T23:36:36.760 INFO:teuthology.orchestra.run.smithi178.stdout: libcephsqlite x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 142 k 2024-02-21T23:36:36.760 INFO:teuthology.orchestra.run.smithi178.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2024-02-21T23:36:36.760 INFO:teuthology.orchestra.run.smithi178.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-02-21T23:36:36.760 INFO:teuthology.orchestra.run.smithi178.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-21T23:36:36.760 INFO:teuthology.orchestra.run.smithi178.stdout: libpq x86_64 13.11-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2024-02-21T23:36:36.760 INFO:teuthology.orchestra.run.smithi178.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-02-21T23:36:36.760 INFO:teuthology.orchestra.run.smithi178.stdout: librabbitmq x86_64 0.9.0-4.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2024-02-21T23:36:36.760 INFO:teuthology.orchestra.run.smithi178.stdout: libradosstriper1 x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 473 k 2024-02-21T23:36:36.761 INFO:teuthology.orchestra.run.smithi178.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-02-21T23:36:36.761 INFO:teuthology.orchestra.run.smithi178.stdout: librgw2 x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 6.5 M 2024-02-21T23:36:36.761 INFO:teuthology.orchestra.run.smithi178.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2024-02-21T23:36:36.761 INFO:teuthology.orchestra.run.smithi178.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2024-02-21T23:36:36.761 INFO:teuthology.orchestra.run.smithi178.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-21T23:36:36.761 INFO:teuthology.orchestra.run.smithi178.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2024-02-21T23:36:36.761 INFO:teuthology.orchestra.run.smithi178.stdout: libzstd-devel x86_64 1.4.4-1.el8 rhel-8-for-x86_64-baseos-rpms 44 k 2024-02-21T23:36:36.761 INFO:teuthology.orchestra.run.smithi178.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-02-21T23:36:36.761 INFO:teuthology.orchestra.run.smithi178.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2024-02-21T23:36:36.761 INFO:teuthology.orchestra.run.smithi178.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-02-21T23:36:36.761 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-02-21T23:36:36.761 INFO:teuthology.orchestra.run.smithi178.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2024-02-21T23:36:36.761 INFO:teuthology.orchestra.run.smithi178.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2024-02-21T23:36:36.762 INFO:teuthology.orchestra.run.smithi178.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-02-21T23:36:36.762 INFO:teuthology.orchestra.run.smithi178.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-02-21T23:36:36.762 INFO:teuthology.orchestra.run.smithi178.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-02-21T23:36:36.762 INFO:teuthology.orchestra.run.smithi178.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-21T23:36:36.762 INFO:teuthology.orchestra.run.smithi178.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-21T23:36:36.762 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-21T23:36:36.762 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-argparse x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 45 k 2024-02-21T23:36:36.762 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-common x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 125 k 2024-02-21T23:36:36.762 INFO:teuthology.orchestra.run.smithi178.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-21T23:36:36.762 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-02-21T23:36:36.762 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-21T23:36:36.762 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-21T23:36:36.762 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cryptography x86_64 3.2.1-6.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2024-02-21T23:36:36.763 INFO:teuthology.orchestra.run.smithi178.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-21T23:36:36.763 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-21T23:36:36.763 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-21T23:36:36.763 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-02-21T23:36:36.763 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-02-21T23:36:36.763 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-02-21T23:36:36.763 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-02-21T23:36:36.763 INFO:teuthology.orchestra.run.smithi178.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-21T23:36:36.763 INFO:teuthology.orchestra.run.smithi178.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-21T23:36:36.763 INFO:teuthology.orchestra.run.smithi178.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-02-21T23:36:36.763 INFO:teuthology.orchestra.run.smithi178.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-02-21T23:36:36.763 INFO:teuthology.orchestra.run.smithi178.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-21T23:36:36.763 INFO:teuthology.orchestra.run.smithi178.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-02-21T23:36:36.763 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-02-21T23:36:36.764 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-02-21T23:36:36.764 INFO:teuthology.orchestra.run.smithi178.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-02-21T23:36:36.764 INFO:teuthology.orchestra.run.smithi178.stdout: python3-packaging noarch 16.8-9.el8 codeready-builder-for-rhel-8-x86_64-rpms 48 k 2024-02-21T23:36:36.764 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-21T23:36:36.764 INFO:teuthology.orchestra.run.smithi178.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-21T23:36:36.764 INFO:teuthology.orchestra.run.smithi178.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-02-21T23:36:36.764 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-02-21T23:36:36.764 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-02-21T23:36:36.764 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-02-21T23:36:36.764 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-02-21T23:36:36.764 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-02-21T23:36:36.765 INFO:teuthology.orchestra.run.smithi178.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-21T23:36:36.765 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-02-21T23:36:36.765 INFO:teuthology.orchestra.run.smithi178.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-21T23:36:36.765 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-21T23:36:36.765 INFO:teuthology.orchestra.run.smithi178.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-02-21T23:36:36.765 INFO:teuthology.orchestra.run.smithi178.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-21T23:36:36.765 INFO:teuthology.orchestra.run.smithi178.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-21T23:36:36.765 INFO:teuthology.orchestra.run.smithi178.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-21T23:36:36.765 INFO:teuthology.orchestra.run.smithi178.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-21T23:36:36.765 INFO:teuthology.orchestra.run.smithi178.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-21T23:36:36.765 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-21T23:36:36.765 INFO:teuthology.orchestra.run.smithi178.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-21T23:36:36.765 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-21T23:36:36.765 INFO:teuthology.orchestra.run.smithi178.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-02-21T23:36:36.766 INFO:teuthology.orchestra.run.smithi178.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-21T23:36:36.766 INFO:teuthology.orchestra.run.smithi178.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-02-21T23:36:36.766 INFO:teuthology.orchestra.run.smithi178.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-02-21T23:36:36.766 INFO:teuthology.orchestra.run.smithi178.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2024-02-21T23:36:36.766 INFO:teuthology.orchestra.run.smithi178.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-02-21T23:36:36.766 INFO:teuthology.orchestra.run.smithi178.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2024-02-21T23:36:36.766 INFO:teuthology.orchestra.run.smithi178.stdout:Enabling module streams: 2024-02-21T23:36:36.766 INFO:teuthology.orchestra.run.smithi178.stdout: postgresql 10 2024-02-21T23:36:36.766 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-21T23:36:36.766 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-02-21T23:36:36.766 INFO:teuthology.orchestra.run.smithi178.stdout:=================================================================================================================================== 2024-02-21T23:36:36.766 INFO:teuthology.orchestra.run.smithi178.stdout:Install 130 Packages 2024-02-21T23:36:36.766 INFO:teuthology.orchestra.run.smithi178.stdout:Upgrade 16 Packages 2024-02-21T23:36:36.767 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-21T23:36:36.767 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 212 M 2024-02-21T23:36:36.767 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-02-21T23:36:37.180 INFO:teuthology.orchestra.run.smithi178.stdout:(1/146): ceph-18.2.1-638.gc1dc18f4.el8.x86_64.r 17 kB/s | 6.4 kB 00:00 2024-02-21T23:36:37.813 INFO:teuthology.orchestra.run.smithi178.stdout:(2/146): ceph-fuse-18.2.1-638.gc1dc18f4.el8.x86 1.3 MB/s | 853 kB 00:00 2024-02-21T23:36:37.960 INFO:teuthology.orchestra.run.smithi178.stdout:(3/146): ceph-immutable-object-cache-18.2.1-638 1.1 MB/s | 165 kB 00:00 2024-02-21T23:36:38.406 INFO:teuthology.orchestra.run.smithi178.stdout:(4/146): ceph-mds-18.2.1-638.gc1dc18f4.el8.x86_ 5.1 MB/s | 2.3 MB 00:00 2024-02-21T23:36:38.698 INFO:teuthology.orchestra.run.smithi178.stdout:(5/146): ceph-mgr-18.2.1-638.gc1dc18f4.el8.x86_ 5.3 MB/s | 1.6 MB 00:00 2024-02-21T23:36:39.181 INFO:teuthology.orchestra.run.smithi178.stdout:(6/146): ceph-mon-18.2.1-638.gc1dc18f4.el8.x86_ 9.6 MB/s | 4.6 MB 00:00 2024-02-21T23:36:39.286 INFO:teuthology.orchestra.run.smithi176.stdout:(10/146): ceph-test-18.2.1-638.gc1dc18f4.el8.x8 15 MB/s | 47 MB 00:03 2024-02-21T23:36:39.381 INFO:teuthology.orchestra.run.smithi176.stdout:(11/146): cephfs-mirror-18.2.1-638.gc1dc18f4.el 2.7 MB/s | 259 kB 00:00 2024-02-21T23:36:39.445 INFO:teuthology.orchestra.run.smithi176.stdout:(12/146): libcephfs-devel-18.2.1-638.gc1dc18f4. 486 kB/s | 31 kB 00:00 2024-02-21T23:36:39.629 INFO:teuthology.orchestra.run.smithi176.stdout:(13/146): ceph-osd-18.2.1-638.gc1dc18f4.el8.x86 4.1 MB/s | 18 MB 00:04 2024-02-21T23:36:39.662 INFO:teuthology.orchestra.run.smithi176.stdout:(14/146): libcephfs2-18.2.1-638.gc1dc18f4.el8.x 3.5 MB/s | 775 kB 00:00 2024-02-21T23:36:39.687 INFO:teuthology.orchestra.run.smithi176.stdout:(15/146): libcephsqlite-18.2.1-638.gc1dc18f4.el 2.4 MB/s | 142 kB 00:00 2024-02-21T23:36:39.704 INFO:teuthology.orchestra.run.smithi176.stdout:(16/146): librados-devel-18.2.1-638.gc1dc18f4.e 3.2 MB/s | 139 kB 00:00 2024-02-21T23:36:39.838 INFO:teuthology.orchestra.run.smithi176.stdout:(17/146): libradosstriper1-18.2.1-638.gc1dc18f4 3.1 MB/s | 473 kB 00:00 2024-02-21T23:36:39.972 INFO:teuthology.orchestra.run.smithi176.stdout:(18/146): python3-ceph-argparse-18.2.1-638.gc1d 334 kB/s | 45 kB 00:00 2024-02-21T23:36:40.080 INFO:teuthology.orchestra.run.smithi176.stdout:(19/146): python3-ceph-common-18.2.1-638.gc1dc1 1.1 MB/s | 125 kB 00:00 2024-02-21T23:36:40.180 INFO:teuthology.orchestra.run.smithi176.stdout:(20/146): python3-cephfs-18.2.1-638.gc1dc18f4.e 1.9 MB/s | 196 kB 00:00 2024-02-21T23:36:40.281 INFO:teuthology.orchestra.run.smithi176.stdout:(21/146): librgw2-18.2.1-638.gc1dc18f4.el8.x86_ 11 MB/s | 6.5 MB 00:00 2024-02-21T23:36:40.339 INFO:teuthology.orchestra.run.smithi176.stdout:(22/146): python3-rados-18.2.1-638.gc1dc18f4.el 2.3 MB/s | 371 kB 00:00 2024-02-21T23:36:40.365 INFO:teuthology.orchestra.run.smithi176.stdout:(23/146): python3-rbd-18.2.1-638.gc1dc18f4.el8. 4.3 MB/s | 364 kB 00:00 2024-02-21T23:36:40.407 INFO:teuthology.orchestra.run.smithi176.stdout:(24/146): python3-rgw-18.2.1-638.gc1dc18f4.el8. 1.6 MB/s | 106 kB 00:00 2024-02-21T23:36:40.432 INFO:teuthology.orchestra.run.smithi176.stdout:(25/146): rbd-fuse-18.2.1-638.gc1dc18f4.el8.x86 1.4 MB/s | 95 kB 00:00 2024-02-21T23:36:40.499 INFO:teuthology.orchestra.run.smithi176.stdout:(26/146): rbd-nbd-18.2.1-638.gc1dc18f4.el8.x86_ 2.7 MB/s | 182 kB 00:00 2024-02-21T23:36:40.566 INFO:teuthology.orchestra.run.smithi176.stdout:(27/146): ceph-grafana-dashboards-18.2.1-638.gc 346 kB/s | 23 kB 00:00 2024-02-21T23:36:40.624 INFO:teuthology.orchestra.run.smithi176.stdout:(28/146): ceph-mgr-cephadm-18.2.1-638.gc1dc18f4 2.3 MB/s | 137 kB 00:00 2024-02-21T23:36:40.783 INFO:teuthology.orchestra.run.smithi176.stdout:(29/146): ceph-mgr-dashboard-18.2.1-638.gc1dc18 11 MB/s | 1.7 MB 00:00 2024-02-21T23:36:40.807 INFO:teuthology.orchestra.run.smithi178.stdout:(7/146): ceph-osd-18.2.1-638.gc1dc18f4.el8.x86_ 11 MB/s | 18 MB 00:01 2024-02-21T23:36:41.142 INFO:teuthology.orchestra.run.smithi176.stdout:(30/146): rbd-mirror-18.2.1-638.gc1dc18f4.el8.x 4.7 MB/s | 3.5 MB 00:00 2024-02-21T23:36:41.242 INFO:teuthology.orchestra.run.smithi176.stdout:(31/146): ceph-mgr-modules-core-18.2.1-638.gc1d 2.4 MB/s | 244 kB 00:00 2024-02-21T23:36:41.342 INFO:teuthology.orchestra.run.smithi176.stdout:(32/146): ceph-mgr-rook-18.2.1-638.gc1dc18f4.el 522 kB/s | 52 kB 00:00 2024-02-21T23:36:41.434 INFO:teuthology.orchestra.run.smithi176.stdout:(33/146): ceph-prometheus-alerts-18.2.1-638.gc1 161 kB/s | 15 kB 00:00 2024-02-21T23:36:41.593 INFO:teuthology.orchestra.run.smithi176.stdout:(34/146): ceph-mgr-diskprediction-local-18.2.1- 9.1 MB/s | 7.4 MB 00:00 2024-02-21T23:36:41.643 INFO:teuthology.orchestra.run.smithi176.stdout:(35/146): ceph-volume-18.2.1-638.gc1dc18f4.el8. 1.2 MB/s | 263 kB 00:00 2024-02-21T23:36:41.685 INFO:teuthology.orchestra.run.smithi176.stdout:(36/146): cephadm-18.2.1-638.gc1dc18f4.el8.noar 1.8 MB/s | 172 kB 00:00 2024-02-21T23:36:41.690 INFO:teuthology.orchestra.run.smithi178.stdout:(8/146): ceph-base-18.2.1-638.gc1dc18f4.el8.x86 1.1 MB/s | 5.4 MB 00:04 2024-02-21T23:36:41.710 INFO:teuthology.orchestra.run.smithi176.stdout:(37/146): cephfs-top-18.2.1-638.gc1dc18f4.el8.n 268 kB/s | 18 kB 00:00 2024-02-21T23:36:41.774 INFO:teuthology.orchestra.run.smithi178.stdout:(9/146): ceph-selinux-18.2.1-638.gc1dc18f4.el8. 293 kB/s | 24 kB 00:00 2024-02-21T23:36:42.049 INFO:teuthology.orchestra.run.smithi178.stdout:(10/146): ceph-radosgw-18.2.1-638.gc1dc18f4.el8 11 MB/s | 13 MB 00:01 2024-02-21T23:36:42.054 INFO:teuthology.orchestra.run.smithi176.stdout:(38/146): liboath-2.6.2-3.el8.x86_64.rpm 171 kB/s | 59 kB 00:00 2024-02-21T23:36:42.116 INFO:teuthology.orchestra.run.smithi178.stdout:(11/146): cephfs-mirror-18.2.1-638.gc1dc18f4.el 3.8 MB/s | 259 kB 00:00 2024-02-21T23:36:42.175 INFO:teuthology.orchestra.run.smithi178.stdout:(12/146): libcephfs-devel-18.2.1-638.gc1dc18f4. 530 kB/s | 31 kB 00:00 2024-02-21T23:36:42.292 INFO:teuthology.orchestra.run.smithi178.stdout:(13/146): libcephfs2-18.2.1-638.gc1dc18f4.el8.x 6.5 MB/s | 775 kB 00:00 2024-02-21T23:36:42.367 INFO:teuthology.orchestra.run.smithi178.stdout:(14/146): libcephsqlite-18.2.1-638.gc1dc18f4.el 1.9 MB/s | 142 kB 00:00 2024-02-21T23:36:42.413 INFO:teuthology.orchestra.run.smithi176.stdout:(39/146): libunwind-1.3.1-3.el8.x86_64.rpm 209 kB/s | 75 kB 00:00 2024-02-21T23:36:42.434 INFO:teuthology.orchestra.run.smithi178.stdout:(15/146): librados-devel-18.2.1-638.gc1dc18f4.e 2.0 MB/s | 139 kB 00:00 2024-02-21T23:36:42.509 INFO:teuthology.orchestra.run.smithi178.stdout:(16/146): libradosstriper1-18.2.1-638.gc1dc18f4 6.2 MB/s | 473 kB 00:00 2024-02-21T23:36:42.938 INFO:teuthology.orchestra.run.smithi176.stdout:(40/146): ceph-common-18.2.1-638.gc1dc18f4.el8. 2.6 MB/s | 22 MB 00:08 2024-02-21T23:36:42.985 INFO:teuthology.orchestra.run.smithi178.stdout:(17/146): librgw2-18.2.1-638.gc1dc18f4.el8.x86_ 14 MB/s | 6.5 MB 00:00 2024-02-21T23:36:42.996 INFO:teuthology.orchestra.run.smithi176.stdout:(41/146): gperftools-libs-2.7-9.el8.x86_64.rpm 234 kB/s | 306 kB 00:01 2024-02-21T23:36:43.043 INFO:teuthology.orchestra.run.smithi178.stdout:(18/146): python3-ceph-argparse-18.2.1-638.gc1d 769 kB/s | 45 kB 00:00 2024-02-21T23:36:43.110 INFO:teuthology.orchestra.run.smithi178.stdout:(19/146): python3-ceph-common-18.2.1-638.gc1dc1 1.8 MB/s | 125 kB 00:00 2024-02-21T23:36:43.230 INFO:teuthology.orchestra.run.smithi176.stdout:(42/146): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 150 kB/s | 44 kB 00:00 2024-02-21T23:36:43.344 INFO:teuthology.orchestra.run.smithi178.stdout:(20/146): ceph-common-18.2.1-638.gc1dc18f4.el8. 3.3 MB/s | 22 MB 00:06 2024-02-21T23:36:43.369 INFO:teuthology.orchestra.run.smithi178.stdout:(21/146): python3-cephfs-18.2.1-638.gc1dc18f4.e 759 kB/s | 196 kB 00:00 2024-02-21T23:36:43.414 INFO:teuthology.orchestra.run.smithi176.stdout:(43/146): python3-cachetools-3.1.1-4.el8.noarch 180 kB/s | 33 kB 00:00 2024-02-21T23:36:43.436 INFO:teuthology.orchestra.run.smithi178.stdout:(22/146): python3-rados-18.2.1-638.gc1dc18f4.el 4.0 MB/s | 371 kB 00:00 2024-02-21T23:36:43.494 INFO:teuthology.orchestra.run.smithi178.stdout:(23/146): python3-rbd-18.2.1-638.gc1dc18f4.el8. 2.9 MB/s | 364 kB 00:00 2024-02-21T23:36:43.520 INFO:teuthology.orchestra.run.smithi178.stdout:(24/146): python3-rgw-18.2.1-638.gc1dc18f4.el8. 1.2 MB/s | 106 kB 00:00 2024-02-21T23:36:43.556 INFO:teuthology.orchestra.run.smithi176.stdout:(44/146): python3-certifi-2018.10.15-7.el8.noar 115 kB/s | 16 kB 00:00 2024-02-21T23:36:43.562 INFO:teuthology.orchestra.run.smithi178.stdout:(25/146): rbd-fuse-18.2.1-638.gc1dc18f4.el8.x86 1.4 MB/s | 95 kB 00:00 2024-02-21T23:36:43.654 INFO:teuthology.orchestra.run.smithi178.stdout:(26/146): rbd-nbd-18.2.1-638.gc1dc18f4.el8.x86_ 1.9 MB/s | 182 kB 00:00 2024-02-21T23:36:43.721 INFO:teuthology.orchestra.run.smithi178.stdout:(27/146): ceph-grafana-dashboards-18.2.1-638.gc 347 kB/s | 23 kB 00:00 2024-02-21T23:36:43.722 INFO:teuthology.orchestra.run.smithi176.stdout:(45/146): python3-beautifulsoup4-4.6.3-2.el8.1. 255 kB/s | 185 kB 00:00 2024-02-21T23:36:43.805 INFO:teuthology.orchestra.run.smithi178.stdout:(28/146): ceph-mgr-cephadm-18.2.1-638.gc1dc18f4 1.6 MB/s | 137 kB 00:00 2024-02-21T23:36:43.980 INFO:teuthology.orchestra.run.smithi178.stdout:(29/146): ceph-mgr-dashboard-18.2.1-638.gc1dc18 10 MB/s | 1.7 MB 00:00 2024-02-21T23:36:44.089 INFO:teuthology.orchestra.run.smithi178.stdout:(30/146): rbd-mirror-18.2.1-638.gc1dc18f4.el8.x 6.1 MB/s | 3.5 MB 00:00 2024-02-21T23:36:44.147 INFO:teuthology.orchestra.run.smithi178.stdout:(31/146): ceph-mgr-modules-core-18.2.1-638.gc1d 4.1 MB/s | 244 kB 00:00 2024-02-21T23:36:44.173 INFO:teuthology.orchestra.run.smithi176.stdout:(46/146): python3-asyncssh-2.7.0-2.el8.noarch.r 270 kB/s | 476 kB 00:01 2024-02-21T23:36:44.189 INFO:teuthology.orchestra.run.smithi178.stdout:(32/146): ceph-mgr-rook-18.2.1-638.gc1dc18f4.el 1.2 MB/s | 52 kB 00:00 2024-02-21T23:36:44.231 INFO:teuthology.orchestra.run.smithi178.stdout:(33/146): ceph-prometheus-alerts-18.2.1-638.gc1 354 kB/s | 15 kB 00:00 2024-02-21T23:36:44.256 INFO:teuthology.orchestra.run.smithi176.stdout:(47/146): python3-cheroot-8.5.2-1.el8.noarch.rp 246 kB/s | 173 kB 00:00 2024-02-21T23:36:44.298 INFO:teuthology.orchestra.run.smithi178.stdout:(34/146): ceph-volume-18.2.1-638.gc1dc18f4.el8. 3.9 MB/s | 263 kB 00:00 2024-02-21T23:36:44.374 INFO:teuthology.orchestra.run.smithi178.stdout:(35/146): cephadm-18.2.1-638.gc1dc18f4.el8.noar 2.2 MB/s | 172 kB 00:00 2024-02-21T23:36:44.399 INFO:teuthology.orchestra.run.smithi176.stdout:(48/146): python3-jaraco-6.2-6.el8.noarch.rpm 78 kB/s | 11 kB 00:00 2024-02-21T23:36:44.424 INFO:teuthology.orchestra.run.smithi178.stdout:(36/146): cephfs-top-18.2.1-638.gc1dc18f4.el8.n 360 kB/s | 18 kB 00:00 2024-02-21T23:36:44.565 INFO:teuthology.orchestra.run.smithi176.stdout:(49/146): python3-jaraco-functools-2.0-4.el8.no 105 kB/s | 18 kB 00:00 2024-02-21T23:36:44.591 INFO:teuthology.orchestra.run.smithi176.stdout:(50/146): python3-google-auth-1.1.1-10.el8.noar 195 kB/s | 81 kB 00:00 2024-02-21T23:36:44.708 INFO:teuthology.orchestra.run.smithi178.stdout:(37/146): ceph-mgr-diskprediction-local-18.2.1- 10 MB/s | 7.4 MB 00:00 2024-02-21T23:36:44.882 INFO:teuthology.orchestra.run.smithi176.stdout:(51/146): python3-logutils-0.3.5-11.el8.noarch. 167 kB/s | 49 kB 00:00 2024-02-21T23:36:44.991 INFO:teuthology.orchestra.run.smithi178.stdout:(38/146): gperftools-libs-2.7-9.el8.x86_64.rpm 540 kB/s | 306 kB 00:00 2024-02-21T23:36:45.033 INFO:teuthology.orchestra.run.smithi178.stdout:(39/146): liboath-2.6.2-3.el8.x86_64.rpm 181 kB/s | 59 kB 00:00 2024-02-21T23:36:45.159 INFO:teuthology.orchestra.run.smithi178.stdout:(40/146): libunwind-1.3.1-3.el8.x86_64.rpm 450 kB/s | 75 kB 00:00 2024-02-21T23:36:45.166 INFO:teuthology.orchestra.run.smithi176.stdout:(52/146): python3-more-itertools-7.2.0-3.el8.no 207 kB/s | 59 kB 00:00 2024-02-21T23:36:45.216 INFO:teuthology.orchestra.run.smithi176.stdout:(53/146): python3-cherrypy-18.4.0-1.el8.noarch. 257 kB/s | 384 kB 00:01 2024-02-21T23:36:45.217 INFO:teuthology.orchestra.run.smithi178.stdout:(41/146): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 750 kB/s | 44 kB 00:00 2024-02-21T23:36:45.367 INFO:teuthology.orchestra.run.smithi178.stdout:(42/146): python3-asyncssh-2.7.0-2.el8.noarch.r 1.4 MB/s | 476 kB 00:00 2024-02-21T23:36:45.393 INFO:teuthology.orchestra.run.smithi178.stdout:(43/146): python3-beautifulsoup4-4.6.3-2.el8.1. 1.0 MB/s | 185 kB 00:00 2024-02-21T23:36:45.426 INFO:teuthology.orchestra.run.smithi178.stdout:(44/146): python3-cachetools-3.1.1-4.el8.noarch 563 kB/s | 33 kB 00:00 2024-02-21T23:36:45.452 INFO:teuthology.orchestra.run.smithi178.stdout:(45/146): python3-certifi-2018.10.15-7.el8.noar 278 kB/s | 16 kB 00:00 2024-02-21T23:36:45.458 INFO:teuthology.orchestra.run.smithi176.stdout:(54/146): python3-natsort-7.1.1-2.el8.noarch.rp 206 kB/s | 60 kB 00:00 2024-02-21T23:36:45.575 INFO:teuthology.orchestra.run.smithi176.stdout:(55/146): python3-portend-2.6-1.el8.noarch.rpm 134 kB/s | 16 kB 00:00 2024-02-21T23:36:45.577 INFO:teuthology.orchestra.run.smithi178.stdout:(46/146): python3-cheroot-8.5.2-1.el8.noarch.rp 1.1 MB/s | 173 kB 00:00 2024-02-21T23:36:45.636 INFO:teuthology.orchestra.run.smithi178.stdout:(47/146): python3-cherrypy-18.4.0-1.el8.noarch. 2.0 MB/s | 384 kB 00:00 2024-02-21T23:36:45.669 INFO:teuthology.orchestra.run.smithi178.stdout:(48/146): python3-google-auth-1.1.1-10.el8.noar 884 kB/s | 81 kB 00:00 2024-02-21T23:36:45.686 INFO:teuthology.orchestra.run.smithi178.stdout:(49/146): python3-jaraco-6.2-6.el8.noarch.rpm 221 kB/s | 11 kB 00:00 2024-02-21T23:36:45.720 INFO:teuthology.orchestra.run.smithi178.stdout:(50/146): python3-jaraco-functools-2.0-4.el8.no 351 kB/s | 18 kB 00:00 2024-02-21T23:36:45.770 INFO:teuthology.orchestra.run.smithi178.stdout:(51/146): python3-logutils-0.3.5-11.el8.noarch. 979 kB/s | 49 kB 00:00 2024-02-21T23:36:45.792 INFO:teuthology.orchestra.run.smithi176.stdout:(56/146): python3-repoze-lru-0.7-6.el8.noarch.r 155 kB/s | 34 kB 00:00 2024-02-21T23:36:45.854 INFO:teuthology.orchestra.run.smithi178.stdout:(52/146): python3-more-itertools-7.2.0-3.el8.no 704 kB/s | 59 kB 00:00 2024-02-21T23:36:45.904 INFO:teuthology.orchestra.run.smithi178.stdout:(53/146): python3-natsort-7.1.1-2.el8.noarch.rp 1.2 MB/s | 60 kB 00:00 2024-02-21T23:36:46.138 INFO:teuthology.orchestra.run.smithi178.stdout:(54/146): python3-pecan-1.3.2-9.el8.noarch.rpm 1.2 MB/s | 283 kB 00:00 2024-02-21T23:36:46.197 INFO:teuthology.orchestra.run.smithi178.stdout:(55/146): python3-portend-2.6-1.el8.noarch.rpm 268 kB/s | 16 kB 00:00 2024-02-21T23:36:46.247 INFO:teuthology.orchestra.run.smithi178.stdout:(56/146): python3-kubernetes-11.0.0-6.el8.noarc 2.1 MB/s | 1.2 MB 00:00 2024-02-21T23:36:46.272 INFO:teuthology.orchestra.run.smithi178.stdout:(57/146): python3-repoze-lru-0.7-6.el8.noarch.r 446 kB/s | 34 kB 00:00 2024-02-21T23:36:46.339 INFO:teuthology.orchestra.run.smithi178.stdout:(58/146): python3-rsa-4.9-2.el8.noarch.rpm 965 kB/s | 64 kB 00:00 2024-02-21T23:36:46.376 INFO:teuthology.orchestra.run.smithi176.stdout:(57/146): python3-pecan-1.3.2-9.el8.noarch.rpm 244 kB/s | 283 kB 00:01 2024-02-21T23:36:46.448 INFO:teuthology.orchestra.run.smithi178.stdout:(59/146): python3-routes-2.4.1-12.el8.noarch.rp 980 kB/s | 196 kB 00:00 2024-02-21T23:36:46.473 INFO:teuthology.orchestra.run.smithi178.stdout:(60/146): python3-simplegeneric-0.8.1-17.el8.no 140 kB/s | 19 kB 00:00 2024-02-21T23:36:46.507 INFO:teuthology.orchestra.run.smithi178.stdout:(61/146): python3-singledispatch-3.4.0.3-18.el8 408 kB/s | 24 kB 00:00 2024-02-21T23:36:46.532 INFO:teuthology.orchestra.run.smithi178.stdout:(62/146): python3-tempora-1.14.1-5.el8.noarch.r 500 kB/s | 29 kB 00:00 2024-02-21T23:36:46.566 INFO:teuthology.orchestra.run.smithi178.stdout:(63/146): python3-trustme-0.6.0-4.el8.noarch.rp 469 kB/s | 27 kB 00:00 2024-02-21T23:36:46.592 INFO:teuthology.orchestra.run.smithi176.stdout:(58/146): python3-routes-2.4.1-12.el8.noarch.rp 245 kB/s | 196 kB 00:00 2024-02-21T23:36:46.666 INFO:teuthology.orchestra.run.smithi178.stdout:(64/146): python3-waitress-1.4.3-1.el8.noarch.r 1.9 MB/s | 255 kB 00:00 2024-02-21T23:36:46.733 INFO:teuthology.orchestra.run.smithi178.stdout:(65/146): python3-websocket-client-0.56.0-5.el8 919 kB/s | 61 kB 00:00 2024-02-21T23:36:46.758 INFO:teuthology.orchestra.run.smithi178.stdout:(66/146): python3-webob-1.8.5-1.el8.1.noarch.rp 1.3 MB/s | 251 kB 00:00 2024-02-21T23:36:46.792 INFO:teuthology.orchestra.run.smithi178.stdout:(67/146): python3-webtest-2.0.33-1.el8.noarch.r 1.4 MB/s | 85 kB 00:00 2024-02-21T23:36:46.826 INFO:teuthology.orchestra.run.smithi178.stdout:(68/146): python3-zc-lockfile-2.0-2.el8.noarch. 342 kB/s | 23 kB 00:00 2024-02-21T23:36:46.851 INFO:teuthology.orchestra.run.smithi178.stdout:(69/146): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.8 MB/s | 70 kB 00:00 2024-02-21T23:36:46.893 INFO:teuthology.orchestra.run.smithi176.stdout:(59/146): python3-rsa-4.9-2.el8.noarch.rpm 124 kB/s | 64 kB 00:00 2024-02-21T23:36:47.018 INFO:teuthology.orchestra.run.smithi178.stdout:(70/146): python3-jsonpointer-1.10-11.el8.noarc 120 kB/s | 20 kB 00:00 2024-02-21T23:36:47.093 INFO:teuthology.orchestra.run.smithi176.stdout:(60/146): python3-simplegeneric-0.8.1-17.el8.no 37 kB/s | 19 kB 00:00 2024-02-21T23:36:47.110 INFO:teuthology.orchestra.run.smithi176.stdout:(61/146): python3-singledispatch-3.4.0.3-18.el8 110 kB/s | 24 kB 00:00 2024-02-21T23:36:47.160 INFO:teuthology.orchestra.run.smithi178.stdout:(71/146): python3-prettytable-0.7.2-14.el8.noar 311 kB/s | 44 kB 00:00 2024-02-21T23:36:47.296 INFO:teuthology.orchestra.run.smithi178.stdout:(72/146): python3-pyasn1-modules-0.3.7-6.el8.no 826 kB/s | 110 kB 00:00 2024-02-21T23:36:47.302 INFO:teuthology.orchestra.run.smithi176.stdout:(62/146): python3-tempora-1.14.1-5.el8.noarch.r 140 kB/s | 29 kB 00:00 2024-02-21T23:36:47.319 INFO:teuthology.orchestra.run.smithi176.stdout:(63/146): python3-trustme-0.6.0-4.el8.noarch.rp 132 kB/s | 27 kB 00:00 2024-02-21T23:36:47.477 INFO:teuthology.orchestra.run.smithi178.stdout:(73/146): python3-jsonpatch-1.21-2.el8.noarch.r 145 kB/s | 27 kB 00:00 2024-02-21T23:36:47.636 INFO:teuthology.orchestra.run.smithi178.stdout:(74/146): python3-pyasn1-0.3.7-6.el8.noarch.rpm 799 kB/s | 126 kB 00:00 2024-02-21T23:36:47.761 INFO:teuthology.orchestra.run.smithi178.stdout:(75/146): thrift-0.13.0-2.el8.x86_64.rpm 1.8 MB/s | 1.7 MB 00:00 2024-02-21T23:36:47.795 INFO:teuthology.orchestra.run.smithi178.stdout:(76/146): python3-werkzeug-0.12.2-4.el8.noarch. 2.8 MB/s | 457 kB 00:00 2024-02-21T23:36:47.970 INFO:teuthology.orchestra.run.smithi178.stdout:(77/146): flex-2.6.1-9.el8.x86_64.rpm 1.8 MB/s | 320 kB 00:00 2024-02-21T23:36:48.020 INFO:teuthology.orchestra.run.smithi178.stdout:(78/146): bison-3.0.4-10.el8.x86_64.rpm 2.6 MB/s | 688 kB 00:00 2024-02-21T23:36:48.054 INFO:teuthology.orchestra.run.smithi178.stdout:(79/146): libtool-2.4.6-25.el8.x86_64.rpm 8.3 MB/s | 709 kB 00:00 2024-02-21T23:36:48.163 INFO:teuthology.orchestra.run.smithi178.stdout:(80/146): python3-markupsafe-0.23-19.el8.x86_64 273 kB/s | 39 kB 00:00 2024-02-21T23:36:48.214 INFO:teuthology.orchestra.run.smithi178.stdout:(81/146): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.6 MB/s | 259 kB 00:00 2024-02-21T23:36:48.264 INFO:teuthology.orchestra.run.smithi178.stdout:(82/146): oniguruma-6.8.2-2.el8.x86_64.rpm 1.8 MB/s | 187 kB 00:00 2024-02-21T23:36:48.277 INFO:teuthology.orchestra.run.smithi176.stdout:(64/146): python3-webob-1.8.5-1.el8.1.noarch.rp 262 kB/s | 251 kB 00:00 2024-02-21T23:36:48.303 INFO:teuthology.orchestra.run.smithi176.stdout:(65/146): python3-waitress-1.4.3-1.el8.noarch.r 255 kB/s | 255 kB 00:01 2024-02-21T23:36:48.322 INFO:teuthology.orchestra.run.smithi178.stdout:(83/146): uuid-1.6.2-43.el8.x86_64.rpm 589 kB/s | 64 kB 00:00 2024-02-21T23:36:48.364 INFO:teuthology.orchestra.run.smithi178.stdout:(84/146): python3-pyOpenSSL-19.0.0-1.el8.noarch 1.0 MB/s | 103 kB 00:00 2024-02-21T23:36:48.553 INFO:teuthology.orchestra.run.smithi176.stdout:(66/146): python3-websocket-client-0.56.0-5.el8 222 kB/s | 61 kB 00:00 2024-02-21T23:36:48.573 INFO:teuthology.orchestra.run.smithi178.stdout:(85/146): python3-babel-2.5.1-7.el8.noarch.rpm 19 MB/s | 4.8 MB 00:00 2024-02-21T23:36:48.607 INFO:teuthology.orchestra.run.smithi178.stdout:(86/146): python3-jinja2-2.10.1-3.el8.noarch.rp 2.2 MB/s | 538 kB 00:00 2024-02-21T23:36:48.645 INFO:teuthology.orchestra.run.smithi176.stdout:(67/146): python3-webtest-2.0.33-1.el8.noarch.r 249 kB/s | 85 kB 00:00 2024-02-21T23:36:48.736 INFO:teuthology.orchestra.run.smithi176.stdout:(68/146): python3-zc-lockfile-2.0-2.el8.noarch. 126 kB/s | 23 kB 00:00 2024-02-21T23:36:48.749 INFO:teuthology.orchestra.run.smithi178.stdout:(87/146): socat-1.7.4.1-1.el8.x86_64.rpm 2.2 MB/s | 323 kB 00:00 2024-02-21T23:36:48.763 INFO:teuthology.orchestra.run.smithi176.stdout:(69/146): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.6 MB/s | 70 kB 00:00 2024-02-21T23:36:48.849 INFO:teuthology.orchestra.run.smithi178.stdout:(88/146): python3-numpy-1.14.3-10.el8.x86_64.rp 14 MB/s | 3.7 MB 00:00 2024-02-21T23:36:48.874 INFO:teuthology.orchestra.run.smithi178.stdout:(89/146): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.8 MB/s | 225 kB 00:00 2024-02-21T23:36:48.896 INFO:teuthology.orchestra.run.smithi176.stdout:(70/146): python3-jsonpointer-1.10-11.el8.noarc 150 kB/s | 20 kB 00:00 2024-02-21T23:36:48.949 INFO:teuthology.orchestra.run.smithi178.stdout:(90/146): librdkafka-0.11.4-3.el8.x86_64.rpm 3.5 MB/s | 354 kB 00:00 2024-02-21T23:36:48.988 INFO:teuthology.orchestra.run.smithi176.stdout:(71/146): python3-kubernetes-11.0.0-6.el8.noarc 270 kB/s | 1.2 MB 00:04 2024-02-21T23:36:49.013 INFO:teuthology.orchestra.run.smithi176.stdout:(72/146): python3-prettytable-0.7.2-14.el8.noar 376 kB/s | 44 kB 00:00 2024-02-21T23:36:49.033 INFO:teuthology.orchestra.run.smithi178.stdout:(91/146): python3-mako-1.0.6-14.el8.noarch.rpm 1.8 MB/s | 157 kB 00:00 2024-02-21T23:36:49.117 INFO:teuthology.orchestra.run.smithi178.stdout:(92/146): openblas-threads-0.3.15-6.el8.x86_64. 21 MB/s | 5.2 MB 00:00 2024-02-21T23:36:49.164 INFO:teuthology.orchestra.run.smithi176.stdout:(73/146): python3-jsonpatch-1.21-2.el8.noarch.r 184 kB/s | 27 kB 00:00 2024-02-21T23:36:49.197 INFO:teuthology.orchestra.run.smithi176.stdout:(74/146): python3-pyasn1-modules-0.3.7-6.el8.no 527 kB/s | 110 kB 00:00 2024-02-21T23:36:49.225 INFO:teuthology.orchestra.run.smithi178.stdout:(93/146): openblas-0.3.15-6.el8.x86_64.rpm 26 MB/s | 5.0 MB 00:00 2024-02-21T23:36:49.247 INFO:teuthology.orchestra.run.smithi176.stdout:(75/146): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.5 MB/s | 126 kB 00:00 2024-02-21T23:36:49.259 INFO:teuthology.orchestra.run.smithi178.stdout:(94/146): automake-1.16.1-8.el8.noarch.rpm 4.9 MB/s | 714 kB 00:00 2024-02-21T23:36:49.339 INFO:teuthology.orchestra.run.smithi176.stdout:(76/146): python3-werkzeug-0.12.2-4.el8.noarch. 3.1 MB/s | 457 kB 00:00 2024-02-21T23:36:49.360 INFO:teuthology.orchestra.run.smithi178.stdout:(95/146): libpq-13.11-1.el8.x86_64.rpm 1.4 MB/s | 198 kB 00:00 2024-02-21T23:36:49.373 INFO:teuthology.orchestra.run.smithi176.stdout:(77/146): bison-3.0.4-10.el8.x86_64.rpm 5.4 MB/s | 688 kB 00:00 2024-02-21T23:36:49.410 INFO:teuthology.orchestra.run.smithi178.stdout:(96/146): jq-1.6-7.el8.x86_64.rpm 1.3 MB/s | 202 kB 00:00 2024-02-21T23:36:49.540 INFO:teuthology.orchestra.run.smithi176.stdout:(78/146): flex-2.6.1-9.el8.x86_64.rpm 1.6 MB/s | 320 kB 00:00 2024-02-21T23:36:49.543 INFO:teuthology.orchestra.run.smithi178.stdout:(97/146): python3-pytz-2017.2-11.el8.noarch.rpm 295 kB/s | 54 kB 00:00 2024-02-21T23:36:49.573 INFO:teuthology.orchestra.run.smithi176.stdout:(79/146): libtool-2.4.6-25.el8.x86_64.rpm 3.5 MB/s | 709 kB 00:00 2024-02-21T23:36:49.627 INFO:teuthology.orchestra.run.smithi178.stdout:(98/146): postgresql-10.23-3.module+el8.9.0+208 6.9 MB/s | 1.5 MB 00:00 2024-02-21T23:36:49.674 INFO:teuthology.orchestra.run.smithi176.stdout:(80/146): python3-markupsafe-0.23-19.el8.x86_64 291 kB/s | 39 kB 00:00 2024-02-21T23:36:49.699 INFO:teuthology.orchestra.run.smithi176.stdout:(81/146): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.0 MB/s | 259 kB 00:00 2024-02-21T23:36:49.702 INFO:teuthology.orchestra.run.smithi178.stdout:(99/146): postgresql-contrib-10.23-3.module+el8 5.0 MB/s | 811 kB 00:00 2024-02-21T23:36:49.791 INFO:teuthology.orchestra.run.smithi176.stdout:(82/146): oniguruma-6.8.2-2.el8.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-02-21T23:36:49.816 INFO:teuthology.orchestra.run.smithi176.stdout:(83/146): uuid-1.6.2-43.el8.x86_64.rpm 547 kB/s | 64 kB 00:00 2024-02-21T23:36:49.883 INFO:teuthology.orchestra.run.smithi176.stdout:(84/146): python3-pyOpenSSL-19.0.0-1.el8.noarch 1.1 MB/s | 103 kB 00:00 2024-02-21T23:36:49.894 INFO:teuthology.orchestra.run.smithi178.stdout:(100/146): postgresql-server-10.23-3.module+el8 19 MB/s | 5.1 MB 00:00 2024-02-21T23:36:49.970 INFO:teuthology.orchestra.run.smithi178.stdout:(101/146): python3-packaging-16.8-9.el8.noarch. 644 kB/s | 48 kB 00:00 2024-02-21T23:36:50.042 INFO:teuthology.orchestra.run.smithi176.stdout:(85/146): python3-babel-2.5.1-7.el8.noarch.rpm 21 MB/s | 4.8 MB 00:00 2024-02-21T23:36:50.076 INFO:teuthology.orchestra.run.smithi176.stdout:(86/146): python3-jinja2-2.10.1-3.el8.noarch.rp 2.7 MB/s | 538 kB 00:00 2024-02-21T23:36:50.212 INFO:teuthology.orchestra.run.smithi178.stdout:(102/146): python3-scipy-1.0.0-21.module+el8.9. 27 MB/s | 14 MB 00:00 2024-02-21T23:36:50.229 INFO:teuthology.orchestra.run.smithi178.stdout:(103/146): libverto-devel-0.3.0-5.el8.x86_64.rp 69 kB/s | 18 kB 00:00 2024-02-21T23:36:50.251 INFO:teuthology.orchestra.run.smithi176.stdout:(87/146): python3-numpy-1.14.3-10.el8.x86_64.rp 18 MB/s | 3.7 MB 00:00 2024-02-21T23:36:50.276 INFO:teuthology.orchestra.run.smithi176.stdout:(88/146): socat-1.7.4.1-1.el8.x86_64.rpm 1.6 MB/s | 323 kB 00:00 2024-02-21T23:36:50.346 INFO:teuthology.orchestra.run.smithi178.stdout:(104/146): python3-jwt-1.6.1-2.el8.noarch.rpm 319 kB/s | 43 kB 00:00 2024-02-21T23:36:50.371 INFO:teuthology.orchestra.run.smithi178.stdout:(105/146): python3-oauthlib-2.1.0-1.el8.noarch. 1.1 MB/s | 155 kB 00:00 2024-02-21T23:36:50.468 INFO:teuthology.orchestra.run.smithi176.stdout:(89/146): librdkafka-0.11.4-3.el8.x86_64.rpm 1.8 MB/s | 354 kB 00:00 2024-02-21T23:36:50.493 INFO:teuthology.orchestra.run.smithi176.stdout:(90/146): python3-numpy-f2py-1.14.3-10.el8.x86_ 930 kB/s | 225 kB 00:00 2024-02-21T23:36:50.496 INFO:teuthology.orchestra.run.smithi178.stdout:(106/146): python3-pycparser-2.14-14.el8.noarch 725 kB/s | 109 kB 00:00 2024-02-21T23:36:50.660 INFO:teuthology.orchestra.run.smithi176.stdout:(91/146): python3-mako-1.0.6-14.el8.noarch.rpm 943 kB/s | 157 kB 00:00 2024-02-21T23:36:50.777 INFO:teuthology.orchestra.run.smithi176.stdout:(92/146): openblas-threads-0.3.15-6.el8.x86_64. 17 MB/s | 5.2 MB 00:00 2024-02-21T23:36:50.886 INFO:teuthology.orchestra.run.smithi176.stdout:(93/146): openblas-0.3.15-6.el8.x86_64.rpm 22 MB/s | 5.0 MB 00:00 2024-02-21T23:36:50.922 INFO:teuthology.orchestra.run.smithi178.stdout:(107/146): python3-requests-oauthlib-1.0.0-1.el 77 kB/s | 43 kB 00:00 2024-02-21T23:36:50.961 INFO:teuthology.orchestra.run.smithi176.stdout:(94/146): automake-1.16.1-8.el8.noarch.rpm 3.8 MB/s | 714 kB 00:00 2024-02-21T23:36:51.023 INFO:teuthology.orchestra.run.smithi176.stdout:(95/146): libpq-13.11-1.el8.x86_64.rpm 1.4 MB/s | 198 kB 00:00 2024-02-21T23:36:51.031 INFO:teuthology.orchestra.run.smithi178.stdout:(108/146): libaio-devel-0.3.112-1.el8.x86_64.rp 35 kB/s | 19 kB 00:00 2024-02-21T23:36:51.056 INFO:teuthology.orchestra.run.smithi178.stdout:(109/146): libzstd-devel-1.4.4-1.el8.x86_64.rpm 328 kB/s | 44 kB 00:00 2024-02-21T23:36:51.079 INFO:teuthology.orchestra.run.smithi176.stdout:(96/146): jq-1.6-7.el8.x86_64.rpm 1.7 MB/s | 202 kB 00:00 2024-02-21T23:36:51.112 INFO:teuthology.orchestra.run.smithi176.stdout:(97/146): python3-pytz-2017.2-11.el8.noarch.rpm 611 kB/s | 54 kB 00:00 2024-02-21T23:36:51.221 INFO:teuthology.orchestra.run.smithi176.stdout:(98/146): postgresql-10.23-3.module+el8.9.0+208 11 MB/s | 1.5 MB 00:00 2024-02-21T23:36:51.223 INFO:teuthology.orchestra.run.smithi178.stdout:(110/146): pcre2-devel-10.32-2.el8.x86_64.rpm 3.1 MB/s | 605 kB 00:00 2024-02-21T23:36:51.248 INFO:teuthology.orchestra.run.smithi178.stdout:(111/146): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.2 MB/s | 229 kB 00:00 2024-02-21T23:36:51.254 INFO:teuthology.orchestra.run.smithi176.stdout:(99/146): postgresql-contrib-10.23-3.module+el8 5.6 MB/s | 811 kB 00:00 2024-02-21T23:36:51.332 INFO:teuthology.orchestra.run.smithi178.stdout:(112/146): pcre2-utf32-10.32-2.el8.x86_64.rpm 2.0 MB/s | 220 kB 00:00 2024-02-21T23:36:51.349 INFO:teuthology.orchestra.run.smithi178.stdout:(113/146): libselinux-devel-2.9-5.el8.x86_64.rp 1.9 MB/s | 200 kB 00:00 2024-02-21T23:36:51.421 INFO:teuthology.orchestra.run.smithi176.stdout:(100/146): postgresql-server-10.23-3.module+el8 25 MB/s | 5.1 MB 00:00 2024-02-21T23:36:51.474 INFO:teuthology.orchestra.run.smithi178.stdout:(114/146): keyutils-libs-devel-1.5.10-9.el8.x86 339 kB/s | 48 kB 00:00 2024-02-21T23:36:51.499 INFO:teuthology.orchestra.run.smithi178.stdout:(115/146): libsepol-devel-2.9-3.el8.x86_64.rpm 579 kB/s | 87 kB 00:00 2024-02-21T23:36:51.546 INFO:teuthology.orchestra.run.smithi176.stdout:(101/146): python3-packaging-16.8-9.el8.noarch. 387 kB/s | 48 kB 00:00 2024-02-21T23:36:51.617 INFO:teuthology.orchestra.run.smithi178.stdout:(116/146): zlib-devel-1.2.11-18.el8_5.x86_64.rp 406 kB/s | 58 kB 00:00 2024-02-21T23:36:51.642 INFO:teuthology.orchestra.run.smithi178.stdout:(117/146): libcom_err-devel-1.45.6-4.el8.x86_64 272 kB/s | 39 kB 00:00 2024-02-21T23:36:51.697 INFO:teuthology.orchestra.run.smithi176.stdout:(102/146): python3-scipy-1.0.0-21.module+el8.9. 31 MB/s | 14 MB 00:00 2024-02-21T23:36:51.722 INFO:teuthology.orchestra.run.smithi176.stdout:(103/146): libverto-devel-0.3.0-5.el8.x86_64.rp 102 kB/s | 18 kB 00:00 2024-02-21T23:36:51.767 INFO:teuthology.orchestra.run.smithi178.stdout:(118/146): fuse-2.9.7-17.el8.x86_64.rpm 553 kB/s | 83 kB 00:00 2024-02-21T23:36:51.831 INFO:teuthology.orchestra.run.smithi176.stdout:(104/146): python3-jwt-1.6.1-2.el8.noarch.rpm 319 kB/s | 43 kB 00:00 2024-02-21T23:36:51.859 INFO:teuthology.orchestra.run.smithi178.stdout:(119/146): libgfortran-8.5.0-20.el8.x86_64.rpm 2.9 MB/s | 645 kB 00:00 2024-02-21T23:36:51.873 INFO:teuthology.orchestra.run.smithi176.stdout:(105/146): python3-oauthlib-2.1.0-1.el8.noarch. 1.0 MB/s | 155 kB 00:00 2024-02-21T23:36:51.884 INFO:teuthology.orchestra.run.smithi178.stdout:(120/146): libquadmath-8.5.0-20.el8.x86_64.rpm 1.4 MB/s | 172 kB 00:00 2024-02-21T23:36:51.965 INFO:teuthology.orchestra.run.smithi176.stdout:(106/146): python3-requests-oauthlib-1.0.0-1.el 465 kB/s | 43 kB 00:00 2024-02-21T23:36:51.993 INFO:teuthology.orchestra.run.smithi178.stdout:(121/146): libuuid-devel-2.32.1-43.el8.x86_64.r 748 kB/s | 100 kB 00:00 2024-02-21T23:36:52.032 INFO:teuthology.orchestra.run.smithi176.stdout:(107/146): python3-pycparser-2.14-14.el8.noarch 544 kB/s | 109 kB 00:00 2024-02-21T23:36:52.123 INFO:teuthology.orchestra.run.smithi176.stdout:(108/146): libaio-devel-0.3.112-1.el8.x86_64.rp 118 kB/s | 19 kB 00:00 2024-02-21T23:36:52.126 INFO:teuthology.orchestra.run.smithi178.stdout:(122/146): python3-cryptography-3.2.1-6.el8.x86 4.1 MB/s | 559 kB 00:00 2024-02-21T23:36:52.152 INFO:teuthology.orchestra.run.smithi178.stdout:(123/146): python3-cffi-1.11.5-6.el8.x86_64.rpm 890 kB/s | 238 kB 00:00 2024-02-21T23:36:52.224 INFO:teuthology.orchestra.run.smithi176.stdout:(109/146): libzstd-devel-1.4.4-1.el8.x86_64.rpm 228 kB/s | 44 kB 00:00 2024-02-21T23:36:52.252 INFO:teuthology.orchestra.run.smithi178.stdout:(124/146): xfsprogs-devel-5.0.0-12.el8.x86_64.r 580 kB/s | 73 kB 00:00 2024-02-21T23:36:52.336 INFO:teuthology.orchestra.run.smithi178.stdout:(125/146): NetworkManager-initscripts-updown-1. 784 kB/s | 144 kB 00:00 2024-02-21T23:36:52.348 INFO:teuthology.orchestra.run.smithi176.stdout:(110/146): pcre2-devel-10.32-2.el8.x86_64.rpm 2.6 MB/s | 605 kB 00:00 2024-02-21T23:36:52.374 INFO:teuthology.orchestra.run.smithi176.stdout:(111/146): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.5 MB/s | 229 kB 00:00 2024-02-21T23:36:52.411 INFO:teuthology.orchestra.run.smithi178.stdout:(126/146): elfutils-libelf-devel-0.189-3.el8.x8 389 kB/s | 62 kB 00:00 2024-02-21T23:36:52.453 INFO:teuthology.orchestra.run.smithi178.stdout:(127/146): krb5-devel-1.18.2-26.el8_9.x86_64.rp 4.7 MB/s | 562 kB 00:00 2024-02-21T23:36:52.500 INFO:teuthology.orchestra.run.smithi176.stdout:(112/146): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.4 MB/s | 220 kB 00:00 2024-02-21T23:36:52.525 INFO:teuthology.orchestra.run.smithi176.stdout:(113/146): libselinux-devel-2.9-5.el8.x86_64.rp 1.3 MB/s | 200 kB 00:00 2024-02-21T23:36:52.528 INFO:teuthology.orchestra.run.smithi178.stdout:(128/146): librabbitmq-0.9.0-4.el8.x86_64.rpm 401 kB/s | 47 kB 00:00 2024-02-21T23:36:52.650 INFO:teuthology.orchestra.run.smithi176.stdout:(114/146): libsepol-devel-2.9-3.el8.x86_64.rpm 696 kB/s | 87 kB 00:00 2024-02-21T23:36:52.679 INFO:teuthology.orchestra.run.smithi178.stdout:(129/146): openssl-devel-1.1.1k-12.el8_9.x86_64 10 MB/s | 2.3 MB 00:00 2024-02-21T23:36:52.692 INFO:teuthology.orchestra.run.smithi176.stdout:(115/146): keyutils-libs-devel-1.5.10-9.el8.x86 250 kB/s | 48 kB 00:00 2024-02-21T23:36:52.784 INFO:teuthology.orchestra.run.smithi176.stdout:(116/146): zlib-devel-1.2.11-18.el8_5.x86_64.rp 432 kB/s | 58 kB 00:00 2024-02-21T23:36:52.843 INFO:teuthology.orchestra.run.smithi176.stdout:(117/146): libcom_err-devel-1.45.6-4.el8.x86_64 257 kB/s | 39 kB 00:00 2024-02-21T23:36:52.951 INFO:teuthology.orchestra.run.smithi176.stdout:(118/146): fuse-2.9.7-17.el8.x86_64.rpm 497 kB/s | 83 kB 00:00 2024-02-21T23:36:53.029 INFO:teuthology.orchestra.run.smithi178.stdout:(130/146): librados2-18.2.1-638.gc1dc18f4.el8.x 7.3 MB/s | 3.7 MB 00:00 2024-02-21T23:36:53.060 INFO:teuthology.orchestra.run.smithi176.stdout:(119/146): libgfortran-8.5.0-20.el8.x86_64.rpm 2.9 MB/s | 645 kB 00:00 2024-02-21T23:36:53.102 INFO:teuthology.orchestra.run.smithi176.stdout:(120/146): libquadmath-8.5.0-20.el8.x86_64.rpm 1.1 MB/s | 172 kB 00:00 2024-02-21T23:36:53.161 INFO:teuthology.orchestra.run.smithi176.stdout:(121/146): libuuid-devel-2.32.1-43.el8.x86_64.r 996 kB/s | 100 kB 00:00 2024-02-21T23:36:53.221 INFO:teuthology.orchestra.run.smithi178.stdout:(131/146): librbd1-18.2.1-638.gc1dc18f4.el8.x86 6.6 MB/s | 3.6 MB 00:00 2024-02-21T23:36:53.246 INFO:teuthology.orchestra.run.smithi178.stdout:(132/146): elfutils-libelf-0.189-3.el8.x86_64.r 1.0 MB/s | 232 kB 00:00 2024-02-21T23:36:53.277 INFO:teuthology.orchestra.run.smithi176.stdout:(122/146): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.3 MB/s | 238 kB 00:00 2024-02-21T23:36:53.353 INFO:teuthology.orchestra.run.smithi176.stdout:(123/146): python3-cryptography-3.2.1-6.el8.x86 2.8 MB/s | 559 kB 00:00 2024-02-21T23:36:53.372 INFO:teuthology.orchestra.run.smithi178.stdout:(133/146): elfutils-libs-0.189-3.el8.x86_64.rpm 2.0 MB/s | 303 kB 00:00 2024-02-21T23:36:53.428 INFO:teuthology.orchestra.run.smithi176.stdout:(124/146): xfsprogs-devel-5.0.0-12.el8.x86_64.r 482 kB/s | 73 kB 00:00 2024-02-21T23:36:53.430 INFO:teuthology.orchestra.run.smithi178.stdout:(134/146): libmount-2.32.1-43.el8.x86_64.rpm 1.3 MB/s | 237 kB 00:00 2024-02-21T23:36:53.545 INFO:teuthology.orchestra.run.smithi176.stdout:(125/146): NetworkManager-initscripts-updown-1. 751 kB/s | 144 kB 00:00 2024-02-21T23:36:53.598 INFO:teuthology.orchestra.run.smithi178.stdout:(135/146): libsmartcols-2.32.1-43.el8.x86_64.rp 797 kB/s | 179 kB 00:00 2024-02-21T23:36:53.604 INFO:teuthology.orchestra.run.smithi176.stdout:(126/146): elfutils-libelf-devel-0.189-3.el8.x8 352 kB/s | 62 kB 00:00 2024-02-21T23:36:53.640 INFO:teuthology.orchestra.run.smithi178.stdout:(136/146): libuuid-2.32.1-43.el8.x86_64.rpm 474 kB/s | 99 kB 00:00 2024-02-21T23:36:53.679 INFO:teuthology.orchestra.run.smithi176.stdout:(127/146): krb5-devel-1.18.2-26.el8_9.x86_64.rp 4.1 MB/s | 562 kB 00:00 2024-02-21T23:36:53.746 INFO:teuthology.orchestra.run.smithi176.stdout:(128/146): librabbitmq-0.9.0-4.el8.x86_64.rpm 331 kB/s | 47 kB 00:00 2024-02-21T23:36:53.806 INFO:teuthology.orchestra.run.smithi178.stdout:(137/146): util-linux-2.32.1-43.el8.x86_64.rpm 12 MB/s | 2.5 MB 00:00 2024-02-21T23:36:53.832 INFO:teuthology.orchestra.run.smithi178.stdout:(138/146): util-linux-user-2.32.1-43.el8.x86_64 535 kB/s | 103 kB 00:00 2024-02-21T23:36:53.919 INFO:teuthology.orchestra.run.smithi176.stdout:(129/146): openssl-devel-1.1.1k-12.el8_9.x86_64 9.7 MB/s | 2.3 MB 00:00 2024-02-21T23:36:53.982 INFO:teuthology.orchestra.run.smithi178.stdout:(139/146): iproute-6.2.0-5.el8_9.x86_64.rpm 4.8 MB/s | 854 kB 00:00 2024-02-21T23:36:54.007 INFO:teuthology.orchestra.run.smithi178.stdout:(140/146): libblkid-2.32.1-43.el8.x86_64.rpm 1.2 MB/s | 220 kB 00:00 2024-02-21T23:36:54.108 INFO:teuthology.orchestra.run.smithi178.stdout:(141/146): libfdisk-2.32.1-43.el8.x86_64.rpm 2.0 MB/s | 254 kB 00:00 2024-02-21T23:36:54.141 INFO:teuthology.orchestra.run.smithi178.stdout:(142/146): openssl-1.1.1k-12.el8_9.x86_64.rpm 5.2 MB/s | 711 kB 00:00 2024-02-21T23:36:54.242 INFO:teuthology.orchestra.run.smithi178.stdout:(143/146): openssl-libs-1.1.1k-12.el8_9.x86_64. 11 MB/s | 1.5 MB 00:00 2024-02-21T23:36:54.263 INFO:teuthology.orchestra.run.smithi176.stdout:(130/146): librbd1-18.2.1-638.gc1dc18f4.el8.x86 10 MB/s | 3.6 MB 00:00 2024-02-21T23:36:54.275 INFO:teuthology.orchestra.run.smithi178.stdout:(144/146): xfsdump-3.1.8-7.el8_9.x86_64.rpm 2.4 MB/s | 333 kB 00:00 2024-02-21T23:36:54.359 INFO:teuthology.orchestra.run.smithi178.stdout:(145/146): xfsprogs-5.0.0-12.el8.x86_64.rpm 9.2 MB/s | 1.1 MB 00:00 2024-02-21T23:36:54.363 INFO:teuthology.orchestra.run.smithi176.stdout:(131/146): elfutils-libelf-0.189-3.el8.x86_64.r 2.3 MB/s | 232 kB 00:00 2024-02-21T23:36:54.496 INFO:teuthology.orchestra.run.smithi176.stdout:(132/146): librados2-18.2.1-638.gc1dc18f4.el8.x 4.9 MB/s | 3.7 MB 00:00 2024-02-21T23:36:54.531 INFO:teuthology.orchestra.run.smithi176.stdout:(133/146): elfutils-libs-0.189-3.el8.x86_64.rpm 1.8 MB/s | 303 kB 00:00 2024-02-21T23:36:54.698 INFO:teuthology.orchestra.run.smithi176.stdout:(134/146): libmount-2.32.1-43.el8.x86_64.rpm 1.2 MB/s | 237 kB 00:00 2024-02-21T23:36:54.723 INFO:teuthology.orchestra.run.smithi176.stdout:(135/146): libsmartcols-2.32.1-43.el8.x86_64.rp 935 kB/s | 179 kB 00:00 2024-02-21T23:36:54.840 INFO:teuthology.orchestra.run.smithi176.stdout:(136/146): libuuid-2.32.1-43.el8.x86_64.rpm 698 kB/s | 99 kB 00:00 2024-02-21T23:36:54.924 INFO:teuthology.orchestra.run.smithi176.stdout:(137/146): util-linux-2.32.1-43.el8.x86_64.rpm 12 MB/s | 2.5 MB 00:00 2024-02-21T23:36:54.965 INFO:teuthology.orchestra.run.smithi176.stdout:(138/146): util-linux-user-2.32.1-43.el8.x86_64 821 kB/s | 103 kB 00:00 2024-02-21T23:36:55.074 INFO:teuthology.orchestra.run.smithi176.stdout:(139/146): iproute-6.2.0-5.el8_9.x86_64.rpm 5.6 MB/s | 854 kB 00:00 2024-02-21T23:36:55.099 INFO:teuthology.orchestra.run.smithi176.stdout:(140/146): libblkid-2.32.1-43.el8.x86_64.rpm 1.6 MB/s | 220 kB 00:00 2024-02-21T23:36:55.250 INFO:teuthology.orchestra.run.smithi176.stdout:(141/146): thrift-0.13.0-2.el8.x86_64.rpm 269 kB/s | 1.7 MB 00:06 2024-02-21T23:36:55.283 INFO:teuthology.orchestra.run.smithi176.stdout:(142/146): libfdisk-2.32.1-43.el8.x86_64.rpm 1.2 MB/s | 254 kB 00:00 2024-02-21T23:36:55.325 INFO:teuthology.orchestra.run.smithi176.stdout:(143/146): openssl-1.1.1k-12.el8_9.x86_64.rpm 3.1 MB/s | 711 kB 00:00 2024-02-21T23:36:55.434 INFO:teuthology.orchestra.run.smithi176.stdout:(144/146): xfsdump-3.1.8-7.el8_9.x86_64.rpm 2.3 MB/s | 333 kB 00:00 2024-02-21T23:36:55.493 INFO:teuthology.orchestra.run.smithi176.stdout:(145/146): openssl-libs-1.1.1k-12.el8_9.x86_64. 6.1 MB/s | 1.5 MB 00:00 2024-02-21T23:36:55.576 INFO:teuthology.orchestra.run.smithi176.stdout:(146/146): xfsprogs-5.0.0-12.el8.x86_64.rpm 4.3 MB/s | 1.1 MB 00:00 2024-02-21T23:36:55.579 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-02-21T23:36:55.579 INFO:teuthology.orchestra.run.smithi176.stdout:Total 10 MB/s | 212 MB 00:21 2024-02-21T23:36:55.801 INFO:teuthology.orchestra.run.smithi178.stdout:(146/146): ceph-test-18.2.1-638.gc1dc18f4.el8.x 3.3 MB/s | 47 MB 00:14 2024-02-21T23:36:55.806 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-02-21T23:36:55.806 INFO:teuthology.orchestra.run.smithi178.stdout:Total 11 MB/s | 212 MB 00:19 2024-02-21T23:36:56.656 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-02-21T23:36:56.886 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-02-21T23:36:57.049 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-02-21T23:36:57.049 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-02-21T23:36:57.282 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-02-21T23:36:57.282 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-02-21T23:36:58.231 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-02-21T23:36:58.241 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-02-21T23:36:58.475 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-02-21T23:36:58.485 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-02-21T23:37:03.927 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-02-21T23:37:04.061 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/1 2024-02-21T23:37:04.317 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-02-21T23:37:04.526 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-02-21T23:37:04.532 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-02-21T23:37:04.662 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/1 2024-02-21T23:37:04.695 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/162 2024-02-21T23:37:04.873 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 3/162 2024-02-21T23:37:04.923 INFO:teuthology.orchestra.run.smithi090.stdout:Extra Packages for Enterprise Linux 254 kB/s | 16 MB 01:05 2024-02-21T23:37:04.930 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-02-21T23:37:04.952 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 3/162 2024-02-21T23:37:05.115 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 4/162 2024-02-21T23:37:05.150 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-02-21T23:37:05.171 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 4/162 2024-02-21T23:37:05.336 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/162 2024-02-21T23:37:05.404 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/162 2024-02-21T23:37:05.510 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 3/162 2024-02-21T23:37:05.586 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 3/162 2024-02-21T23:37:05.614 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 6/162 2024-02-21T23:37:05.680 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 6/162 2024-02-21T23:37:05.769 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 4/162 2024-02-21T23:37:05.838 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 4/162 2024-02-21T23:37:05.867 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/162 2024-02-21T23:37:06.081 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-02-21T23:37:06.092 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/162 2024-02-21T23:37:06.260 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 6/162 2024-02-21T23:37:06.350 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 6/162 2024-02-21T23:37:06.471 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-02-21T23:37:06.537 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/162 2024-02-21T23:37:06.689 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : librados2-2:18.2.1-638.gc1dc18f4.el8.x86_64 9/162 2024-02-21T23:37:06.760 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-02-21T23:37:07.077 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librados2-2:18.2.1-638.gc1dc18f4.el8.x86_64 9/162 2024-02-21T23:37:07.171 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-02-21T23:37:07.264 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : librbd1-2:18.2.1-638.gc1dc18f4.el8.x86_64 10/162 2024-02-21T23:37:07.387 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librbd1-2:18.2.1-638.gc1dc18f4.el8.x86_64 10/162 2024-02-21T23:37:07.402 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : librados2-2:18.2.1-638.gc1dc18f4.el8.x86_64 9/162 2024-02-21T23:37:07.573 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcephfs2-2:18.2.1-638.gc1dc18f4.el8.x86_64 11/162 2024-02-21T23:37:07.667 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libcephfs2-2:18.2.1-638.gc1dc18f4.el8.x86_64 11/162 2024-02-21T23:37:07.793 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados2-2:18.2.1-638.gc1dc18f4.el8.x86_64 9/162 2024-02-21T23:37:07.983 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rados-2:18.2.1-638.gc1dc18f4.el8.x86_64 12/162 2024-02-21T23:37:07.985 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : librbd1-2:18.2.1-638.gc1dc18f4.el8.x86_64 10/162 2024-02-21T23:37:08.113 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librbd1-2:18.2.1-638.gc1dc18f4.el8.x86_64 10/162 2024-02-21T23:37:08.285 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephfs2-2:18.2.1-638.gc1dc18f4.el8.x86_64 11/162 2024-02-21T23:37:08.384 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephfs2-2:18.2.1-638.gc1dc18f4.el8.x86_64 11/162 2024-02-21T23:37:08.457 INFO:teuthology.orchestra.run.smithi090.stdout:lab-extras 568 kB/s | 24 kB 00:00 2024-02-21T23:37:08.472 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-02-21T23:37:08.559 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-02-21T23:37:08.695 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rados-2:18.2.1-638.gc1dc18f4.el8.x86_64 12/162 2024-02-21T23:37:08.755 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libpq-13.11-1.el8.x86_64 14/162 2024-02-21T23:37:08.985 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-02-21T23:37:09.006 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 15/162 2024-02-21T23:37:09.059 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-02-21T23:37:09.194 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 16/162 2024-02-21T23:37:09.226 INFO:teuthology.orchestra.run.smithi090.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 13 MB/s | 8.8 MB 00:00 2024-02-21T23:37:09.296 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libpq-13.11-1.el8.x86_64 14/162 2024-02-21T23:37:09.515 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 15/162 2024-02-21T23:37:09.559 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 17/162 2024-02-21T23:37:09.696 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 16/162 2024-02-21T23:37:09.793 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : postgresql-10.23-3.module+el8.9.0+20831+78489779 18/162 2024-02-21T23:37:09.939 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcephsqlite-2:18.2.1-638.gc1dc18f4.el8.x86_64 19/162 2024-02-21T23:37:10.043 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libcephsqlite-2:18.2.1-638.gc1dc18f4.el8.x86_64 19/162 2024-02-21T23:37:10.048 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 17/162 2024-02-21T23:37:10.181 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libradosstriper1-2:18.2.1-638.gc1dc18f4.el8.x86_ 20/162 2024-02-21T23:37:10.256 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libradosstriper1-2:18.2.1-638.gc1dc18f4.el8.x86_ 20/162 2024-02-21T23:37:10.289 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : postgresql-10.23-3.module+el8.9.0+20831+78489779 18/162 2024-02-21T23:37:10.405 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 21/162 2024-02-21T23:37:10.426 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephsqlite-2:18.2.1-638.gc1dc18f4.el8.x86_64 19/162 2024-02-21T23:37:10.536 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 21/162 2024-02-21T23:37:10.561 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephsqlite-2:18.2.1-638.gc1dc18f4.el8.x86_64 19/162 2024-02-21T23:37:10.685 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libradosstriper1-2:18.2.1-638.gc1dc18f4.el8.x86_ 20/162 2024-02-21T23:37:10.726 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 22/162 2024-02-21T23:37:10.780 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libradosstriper1-2:18.2.1-638.gc1dc18f4.el8.x86_ 20/162 2024-02-21T23:37:10.951 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 21/162 2024-02-21T23:37:11.078 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 21/162 2024-02-21T23:37:11.220 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 22/162 2024-02-21T23:37:11.348 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 22/162 2024-02-21T23:37:11.505 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 23/162 2024-02-21T23:37:11.850 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 22/162 2024-02-21T23:37:12.068 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 23/162 2024-02-21T23:37:12.114 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 23/162 2024-02-21T23:37:12.228 INFO:teuthology.orchestra.run.smithi090.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 44 MB/s | 65 MB 00:01 2024-02-21T23:37:12.255 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : openblas-0.3.15-6.el8.x86_64 24/162 2024-02-21T23:37:12.657 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 23/162 2024-02-21T23:37:12.859 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : openblas-0.3.15-6.el8.x86_64 24/162 2024-02-21T23:37:12.963 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 24/162 2024-02-21T23:37:13.211 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 25/162 2024-02-21T23:37:13.360 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 26/162 2024-02-21T23:37:13.538 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 24/162 2024-02-21T23:37:13.558 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 27/162 2024-02-21T23:37:13.840 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 25/162 2024-02-21T23:37:13.848 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/162 2024-02-21T23:37:13.982 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 26/162 2024-02-21T23:37:14.135 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 29/162 2024-02-21T23:37:14.155 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 27/162 2024-02-21T23:37:14.373 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 30/162 2024-02-21T23:37:14.463 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/162 2024-02-21T23:37:14.559 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-ceph-common-2:18.2.1-638.gc1dc18f4.el8.x 31/162 2024-02-21T23:37:14.708 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-ceph-argparse-2:18.2.1-638.gc1dc18f4.el8 32/162 2024-02-21T23:37:14.767 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 29/162 2024-02-21T23:37:14.916 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cephfs-2:18.2.1-638.gc1dc18f4.el8.x86_64 33/162 2024-02-21T23:37:15.036 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 30/162 2024-02-21T23:37:15.271 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-ceph-common-2:18.2.1-638.gc1dc18f4.el8.x 31/162 2024-02-21T23:37:15.412 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-ceph-argparse-2:18.2.1-638.gc1dc18f4.el8 32/162 2024-02-21T23:37:15.692 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cephfs-2:18.2.1-638.gc1dc18f4.el8.x86_64 33/162 2024-02-21T23:37:15.849 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 34/162 2024-02-21T23:37:16.176 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 35/162 2024-02-21T23:37:16.477 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 36/162 2024-02-21T23:37:16.662 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 37/162 2024-02-21T23:37:16.667 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 34/162 2024-02-21T23:37:17.013 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 35/162 2024-02-21T23:37:17.325 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 36/162 2024-02-21T23:37:17.541 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 37/162 2024-02-21T23:37:18.408 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 37/162 2024-02-21T23:37:18.721 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 38/162 2024-02-21T23:37:18.949 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rsa-4.9-2.el8.noarch 39/162 2024-02-21T23:37:19.173 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 40/162 2024-02-21T23:37:19.287 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 37/162 2024-02-21T23:37:19.385 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rbd-2:18.2.1-638.gc1dc18f4.el8.x86_64 41/162 2024-02-21T23:37:19.561 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librados-devel-2:18.2.1-638.gc1dc18f4.el8.x86_64 42/162 2024-02-21T23:37:19.617 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 38/162 2024-02-21T23:37:19.745 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 43/162 2024-02-21T23:37:19.807 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 43/162 2024-02-21T23:37:19.828 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rsa-4.9-2.el8.noarch 39/162 2024-02-21T23:37:20.029 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 44/162 2024-02-21T23:37:20.076 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 40/162 2024-02-21T23:37:20.099 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 44/162 2024-02-21T23:37:20.306 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rbd-2:18.2.1-638.gc1dc18f4.el8.x86_64 41/162 2024-02-21T23:37:20.307 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 45/162 2024-02-21T23:37:20.517 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librados-devel-2:18.2.1-638.gc1dc18f4.el8.x86_64 42/162 2024-02-21T23:37:20.539 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 46/162 2024-02-21T23:37:20.699 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 43/162 2024-02-21T23:37:20.720 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-02-21T23:37:20.797 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 43/162 2024-02-21T23:37:20.999 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 44/162 2024-02-21T23:37:21.073 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 44/162 2024-02-21T23:37:21.171 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-02-21T23:37:21.280 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 45/162 2024-02-21T23:37:21.446 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 48/162 2024-02-21T23:37:21.463 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 48/162 2024-02-21T23:37:21.494 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 46/162 2024-02-21T23:37:21.691 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : fuse-2.9.7-17.el8.x86_64 49/162 2024-02-21T23:37:21.691 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-02-21T23:37:21.867 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 50/162 2024-02-21T23:37:22.033 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 51/162 2024-02-21T23:37:22.124 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-02-21T23:37:22.266 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 52/162 2024-02-21T23:37:22.457 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 53/162 2024-02-21T23:37:22.458 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 48/162 2024-02-21T23:37:22.475 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 48/162 2024-02-21T23:37:22.695 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : fuse-2.9.7-17.el8.x86_64 49/162 2024-02-21T23:37:22.732 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 54/162 2024-02-21T23:37:22.913 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 50/162 2024-02-21T23:37:22.983 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 55/162 2024-02-21T23:37:23.078 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 51/162 2024-02-21T23:37:23.179 INFO:teuthology.orchestra.run.smithi090.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 38 MB/s | 59 MB 00:01 2024-02-21T23:37:23.183 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 56/162 2024-02-21T23:37:23.314 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 52/162 2024-02-21T23:37:23.349 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 57/162 2024-02-21T23:37:23.536 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 53/162 2024-02-21T23:37:23.539 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 58/162 2024-02-21T23:37:23.820 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 54/162 2024-02-21T23:37:23.861 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 59/162 2024-02-21T23:37:24.114 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 55/162 2024-02-21T23:37:24.156 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 60/162 2024-02-21T23:37:24.338 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 61/162 2024-02-21T23:37:24.379 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 56/162 2024-02-21T23:37:24.553 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 57/162 2024-02-21T23:37:24.610 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 62/162 2024-02-21T23:37:24.784 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 58/162 2024-02-21T23:37:24.866 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 63/162 2024-02-21T23:37:25.171 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 59/162 2024-02-21T23:37:25.211 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 64/162 2024-02-21T23:37:25.411 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 65/162 2024-02-21T23:37:25.493 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 60/162 2024-02-21T23:37:25.604 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 66/162 2024-02-21T23:37:25.708 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 61/162 2024-02-21T23:37:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 67/162 2024-02-21T23:37:25.967 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 68/162 2024-02-21T23:37:25.985 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 62/162 2024-02-21T23:37:26.229 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : krb5-devel-1.18.2-26.el8_9.x86_64 69/162 2024-02-21T23:37:26.269 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 63/162 2024-02-21T23:37:26.496 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-packaging-16.8-9.el8.noarch 70/162 2024-02-21T23:37:26.623 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 64/162 2024-02-21T23:37:26.703 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : automake-1.16.1-8.el8.noarch 71/162 2024-02-21T23:37:26.856 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 65/162 2024-02-21T23:37:26.878 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : uuid-1.6.2-43.el8.x86_64 72/162 2024-02-21T23:37:26.960 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 72/162 2024-02-21T23:37:27.041 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 66/162 2024-02-21T23:37:27.178 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 73/162 2024-02-21T23:37:27.217 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 67/162 2024-02-21T23:37:27.260 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 73/162 2024-02-21T23:37:27.486 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : jq-1.6-7.el8.x86_64 74/162 2024-02-21T23:37:27.488 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 68/162 2024-02-21T23:37:27.660 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 75/162 2024-02-21T23:37:27.741 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : krb5-devel-1.18.2-26.el8_9.x86_64 69/162 2024-02-21T23:37:27.843 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 76/162 2024-02-21T23:37:28.038 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-packaging-16.8-9.el8.noarch 70/162 2024-02-21T23:37:28.054 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 77/162 2024-02-21T23:37:28.203 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 78/162 2024-02-21T23:37:28.274 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : automake-1.16.1-8.el8.noarch 71/162 2024-02-21T23:37:28.400 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 79/162 2024-02-21T23:37:28.423 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : uuid-1.6.2-43.el8.x86_64 72/162 2024-02-21T23:37:28.537 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 72/162 2024-02-21T23:37:28.733 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 80/162 2024-02-21T23:37:28.781 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 73/162 2024-02-21T23:37:28.861 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 73/162 2024-02-21T23:37:28.968 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 81/162 2024-02-21T23:37:29.073 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : jq-1.6-7.el8.x86_64 74/162 2024-02-21T23:37:29.222 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 75/162 2024-02-21T23:37:29.233 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 82/162 2024-02-21T23:37:29.397 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 83/162 2024-02-21T23:37:29.414 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 76/162 2024-02-21T23:37:29.566 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 77/162 2024-02-21T23:37:29.625 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 84/162 2024-02-21T23:37:29.765 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 78/162 2024-02-21T23:37:29.832 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 85/162 2024-02-21T23:37:29.970 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 79/162 2024-02-21T23:37:30.056 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 86/162 2024-02-21T23:37:30.245 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 80/162 2024-02-21T23:37:30.272 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 87/162 2024-02-21T23:37:30.441 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 88/162 2024-02-21T23:37:30.463 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 81/162 2024-02-21T23:37:30.657 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 89/162 2024-02-21T23:37:30.677 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 82/162 2024-02-21T23:37:30.884 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 83/162 2024-02-21T23:37:30.897 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 90/162 2024-02-21T23:37:31.066 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 91/162 2024-02-21T23:37:31.079 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 84/162 2024-02-21T23:37:31.318 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 85/162 2024-02-21T23:37:31.331 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-portend-2.6-1.el8.noarch 92/162 2024-02-21T23:37:31.575 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 86/162 2024-02-21T23:37:31.576 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 93/162 2024-02-21T23:37:31.767 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 94/162 2024-02-21T23:37:31.784 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 87/162 2024-02-21T23:37:31.928 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 88/162 2024-02-21T23:37:32.035 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 95/162 2024-02-21T23:37:32.176 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 89/162 2024-02-21T23:37:32.433 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 90/162 2024-02-21T23:37:32.653 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 91/162 2024-02-21T23:37:32.908 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-portend-2.6-1.el8.noarch 92/162 2024-02-21T23:37:33.096 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 96/162 2024-02-21T23:37:33.146 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 93/162 2024-02-21T23:37:33.362 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 94/162 2024-02-21T23:37:33.397 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 97/162 2024-02-21T23:37:33.621 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 95/162 2024-02-21T23:37:33.625 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 98/162 2024-02-21T23:37:33.943 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 99/162 2024-02-21T23:37:34.169 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 100/162 2024-02-21T23:37:34.427 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 101/162 2024-02-21T23:37:34.748 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 96/162 2024-02-21T23:37:35.092 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 97/162 2024-02-21T23:37:35.177 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 102/162 2024-02-21T23:37:35.318 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librgw2-2:18.2.1-638.gc1dc18f4.el8.x86_64 103/162 2024-02-21T23:37:35.338 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 98/162 2024-02-21T23:37:35.403 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librgw2-2:18.2.1-638.gc1dc18f4.el8.x86_64 103/162 2024-02-21T23:37:35.569 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rgw-2:18.2.1-638.gc1dc18f4.el8.x86_64 104/162 2024-02-21T23:37:35.664 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 99/162 2024-02-21T23:37:35.947 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 100/162 2024-02-21T23:37:36.205 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 101/162 2024-02-21T23:37:36.956 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 102/162 2024-02-21T23:37:37.138 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librgw2-2:18.2.1-638.gc1dc18f4.el8.x86_64 103/162 2024-02-21T23:37:37.207 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librgw2-2:18.2.1-638.gc1dc18f4.el8.x86_64 103/162 2024-02-21T23:37:37.413 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rgw-2:18.2.1-638.gc1dc18f4.el8.x86_64 104/162 2024-02-21T23:37:38.378 INFO:teuthology.orchestra.run.smithi090.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-02-21T23:37:38.380 INFO:teuthology.orchestra.run.smithi090.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-02-21T23:37:38.397 INFO:teuthology.orchestra.run.smithi090.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2024-02-21T23:37:38.399 INFO:teuthology.orchestra.run.smithi090.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2024-02-21T23:37:38.400 INFO:teuthology.orchestra.run.smithi090.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2024-02-21T23:37:38.401 INFO:teuthology.orchestra.run.smithi090.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-02-21T23:37:38.404 INFO:teuthology.orchestra.run.smithi090.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-02-21T23:37:38.408 INFO:teuthology.orchestra.run.smithi090.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-02-21T23:37:38.410 INFO:teuthology.orchestra.run.smithi090.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-02-21T23:37:38.622 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-21T23:37:38.636 INFO:teuthology.orchestra.run.smithi090.stdout:=================================================================================================================================== 2024-02-21T23:37:38.636 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-02-21T23:37:38.637 INFO:teuthology.orchestra.run.smithi090.stdout:=================================================================================================================================== 2024-02-21T23:37:38.637 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-21T23:37:38.637 INFO:teuthology.orchestra.run.smithi090.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-13.el8_9 rhel-8-for-x86_64-baseos-rpms 144 k 2024-02-21T23:37:38.637 INFO:teuthology.orchestra.run.smithi090.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2024-02-21T23:37:38.637 INFO:teuthology.orchestra.run.smithi090.stdout: ceph x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 6.4 k 2024-02-21T23:37:38.637 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 5.4 M 2024-02-21T23:37:38.637 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-fuse x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 853 k 2024-02-21T23:37:38.637 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-immutable-object-cache x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 165 k 2024-02-21T23:37:38.637 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 1.6 M 2024-02-21T23:37:38.637 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-cephadm noarch 2:18.2.1-638.gc1dc18f4.el8 ceph-noarch 137 k 2024-02-21T23:37:38.637 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-dashboard noarch 2:18.2.1-638.gc1dc18f4.el8 ceph-noarch 1.7 M 2024-02-21T23:37:38.638 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.1-638.gc1dc18f4.el8 ceph-noarch 7.4 M 2024-02-21T23:37:38.638 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-rook noarch 2:18.2.1-638.gc1dc18f4.el8 ceph-noarch 52 k 2024-02-21T23:37:38.638 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-radosgw x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 13 M 2024-02-21T23:37:38.638 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-test x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 47 M 2024-02-21T23:37:38.638 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-volume noarch 2:18.2.1-638.gc1dc18f4.el8 ceph-noarch 263 k 2024-02-21T23:37:38.638 INFO:teuthology.orchestra.run.smithi090.stdout: cephadm noarch 2:18.2.1-638.gc1dc18f4.el8 ceph-noarch 172 k 2024-02-21T23:37:38.638 INFO:teuthology.orchestra.run.smithi090.stdout: cephfs-mirror x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 259 k 2024-02-21T23:37:38.638 INFO:teuthology.orchestra.run.smithi090.stdout: cephfs-top noarch 2:18.2.1-638.gc1dc18f4.el8 ceph-noarch 18 k 2024-02-21T23:37:38.638 INFO:teuthology.orchestra.run.smithi090.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2024-02-21T23:37:38.638 INFO:teuthology.orchestra.run.smithi090.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2024-02-21T23:37:38.638 INFO:teuthology.orchestra.run.smithi090.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2024-02-21T23:37:38.638 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs-devel x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 31 k 2024-02-21T23:37:38.638 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs2 x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 775 k 2024-02-21T23:37:38.639 INFO:teuthology.orchestra.run.smithi090.stdout: librados-devel x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 139 k 2024-02-21T23:37:38.639 INFO:teuthology.orchestra.run.smithi090.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2024-02-21T23:37:38.639 INFO:teuthology.orchestra.run.smithi090.stdout: libuuid-devel x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 100 k 2024-02-21T23:37:38.639 INFO:teuthology.orchestra.run.smithi090.stdout: openssl-devel x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 2.3 M 2024-02-21T23:37:38.639 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 1.5 M 2024-02-21T23:37:38.639 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-contrib x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 811 k 2024-02-21T23:37:38.639 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-server x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 5.1 M 2024-02-21T23:37:38.639 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cephfs x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 196 k 2024-02-21T23:37:38.639 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 371 k 2024-02-21T23:37:38.639 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rbd x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 364 k 2024-02-21T23:37:38.640 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 106 k 2024-02-21T23:37:38.640 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-fuse x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 95 k 2024-02-21T23:37:38.640 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-mirror x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 3.5 M 2024-02-21T23:37:38.640 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-nbd x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 182 k 2024-02-21T23:37:38.640 INFO:teuthology.orchestra.run.smithi090.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 73 k 2024-02-21T23:37:38.640 INFO:teuthology.orchestra.run.smithi090.stdout:Upgrading: 2024-02-21T23:37:38.640 INFO:teuthology.orchestra.run.smithi090.stdout: elfutils-libelf x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 232 k 2024-02-21T23:37:38.640 INFO:teuthology.orchestra.run.smithi090.stdout: elfutils-libs x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 303 k 2024-02-21T23:37:38.640 INFO:teuthology.orchestra.run.smithi090.stdout: iproute x86_64 6.2.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 854 k 2024-02-21T23:37:38.640 INFO:teuthology.orchestra.run.smithi090.stdout: libblkid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-02-21T23:37:38.640 INFO:teuthology.orchestra.run.smithi090.stdout: libfdisk x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 254 k 2024-02-21T23:37:38.640 INFO:teuthology.orchestra.run.smithi090.stdout: libmount x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 237 k 2024-02-21T23:37:38.641 INFO:teuthology.orchestra.run.smithi090.stdout: librados2 x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 3.7 M 2024-02-21T23:37:38.641 INFO:teuthology.orchestra.run.smithi090.stdout: librbd1 x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 3.6 M 2024-02-21T23:37:38.641 INFO:teuthology.orchestra.run.smithi090.stdout: libsmartcols x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 179 k 2024-02-21T23:37:38.641 INFO:teuthology.orchestra.run.smithi090.stdout: libuuid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 99 k 2024-02-21T23:37:38.641 INFO:teuthology.orchestra.run.smithi090.stdout: openssl x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 711 k 2024-02-21T23:37:38.641 INFO:teuthology.orchestra.run.smithi090.stdout: openssl-libs x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 1.5 M 2024-02-21T23:37:38.641 INFO:teuthology.orchestra.run.smithi090.stdout: util-linux x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 2.5 M 2024-02-21T23:37:38.641 INFO:teuthology.orchestra.run.smithi090.stdout: util-linux-user x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 103 k 2024-02-21T23:37:38.641 INFO:teuthology.orchestra.run.smithi090.stdout: xfsdump x86_64 3.1.8-7.el8_9 rhel-8-for-x86_64-baseos-rpms 333 k 2024-02-21T23:37:38.641 INFO:teuthology.orchestra.run.smithi090.stdout: xfsprogs x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M 2024-02-21T23:37:38.642 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-02-21T23:37:38.642 INFO:teuthology.orchestra.run.smithi090.stdout: automake noarch 1.16.1-8.el8 rhel-8-for-x86_64-appstream-rpms 714 k 2024-02-21T23:37:38.642 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-common x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 22 M 2024-02-21T23:37:38.642 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-grafana-dashboards noarch 2:18.2.1-638.gc1dc18f4.el8 ceph-noarch 23 k 2024-02-21T23:37:38.642 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mds x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 2.3 M 2024-02-21T23:37:38.642 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-modules-core noarch 2:18.2.1-638.gc1dc18f4.el8 ceph-noarch 244 k 2024-02-21T23:37:38.642 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mon x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 4.6 M 2024-02-21T23:37:38.642 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-osd x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 18 M 2024-02-21T23:37:38.642 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-prometheus-alerts noarch 2:18.2.1-638.gc1dc18f4.el8 ceph-noarch 15 k 2024-02-21T23:37:38.643 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-selinux x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 24 k 2024-02-21T23:37:38.643 INFO:teuthology.orchestra.run.smithi090.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-02-21T23:37:38.643 INFO:teuthology.orchestra.run.smithi090.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-21T23:37:38.643 INFO:teuthology.orchestra.run.smithi090.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-02-21T23:37:38.643 INFO:teuthology.orchestra.run.smithi090.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2024-02-21T23:37:38.643 INFO:teuthology.orchestra.run.smithi090.stdout: krb5-devel x86_64 1.18.2-26.el8_9 rhel-8-for-x86_64-baseos-rpms 562 k 2024-02-21T23:37:38.643 INFO:teuthology.orchestra.run.smithi090.stdout: libcephsqlite x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 142 k 2024-02-21T23:37:38.643 INFO:teuthology.orchestra.run.smithi090.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2024-02-21T23:37:38.643 INFO:teuthology.orchestra.run.smithi090.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-02-21T23:37:38.644 INFO:teuthology.orchestra.run.smithi090.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-21T23:37:38.644 INFO:teuthology.orchestra.run.smithi090.stdout: libpq x86_64 13.11-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2024-02-21T23:37:38.644 INFO:teuthology.orchestra.run.smithi090.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-02-21T23:37:38.644 INFO:teuthology.orchestra.run.smithi090.stdout: librabbitmq x86_64 0.9.0-4.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2024-02-21T23:37:38.644 INFO:teuthology.orchestra.run.smithi090.stdout: libradosstriper1 x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 473 k 2024-02-21T23:37:38.644 INFO:teuthology.orchestra.run.smithi090.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-02-21T23:37:38.644 INFO:teuthology.orchestra.run.smithi090.stdout: librgw2 x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 6.5 M 2024-02-21T23:37:38.644 INFO:teuthology.orchestra.run.smithi090.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2024-02-21T23:37:38.644 INFO:teuthology.orchestra.run.smithi090.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2024-02-21T23:37:38.644 INFO:teuthology.orchestra.run.smithi090.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-21T23:37:38.644 INFO:teuthology.orchestra.run.smithi090.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2024-02-21T23:37:38.644 INFO:teuthology.orchestra.run.smithi090.stdout: libzstd-devel x86_64 1.4.4-1.el8 rhel-8-for-x86_64-baseos-rpms 44 k 2024-02-21T23:37:38.645 INFO:teuthology.orchestra.run.smithi090.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-02-21T23:37:38.645 INFO:teuthology.orchestra.run.smithi090.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2024-02-21T23:37:38.645 INFO:teuthology.orchestra.run.smithi090.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-02-21T23:37:38.645 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-02-21T23:37:38.645 INFO:teuthology.orchestra.run.smithi090.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2024-02-21T23:37:38.645 INFO:teuthology.orchestra.run.smithi090.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2024-02-21T23:37:38.645 INFO:teuthology.orchestra.run.smithi090.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-02-21T23:37:38.645 INFO:teuthology.orchestra.run.smithi090.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-02-21T23:37:38.645 INFO:teuthology.orchestra.run.smithi090.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-02-21T23:37:38.645 INFO:teuthology.orchestra.run.smithi090.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-21T23:37:38.645 INFO:teuthology.orchestra.run.smithi090.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-21T23:37:38.645 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-21T23:37:38.645 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-argparse x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 45 k 2024-02-21T23:37:38.646 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-common x86_64 2:18.2.1-638.gc1dc18f4.el8 ceph 125 k 2024-02-21T23:37:38.646 INFO:teuthology.orchestra.run.smithi090.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-21T23:37:38.646 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-02-21T23:37:38.646 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-21T23:37:38.646 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-21T23:37:38.646 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cryptography x86_64 3.2.1-6.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2024-02-21T23:37:38.646 INFO:teuthology.orchestra.run.smithi090.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-21T23:37:38.646 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-21T23:37:38.646 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-21T23:37:38.646 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-02-21T23:37:38.646 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-02-21T23:37:38.646 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-02-21T23:37:38.646 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-02-21T23:37:38.646 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-21T23:37:38.647 INFO:teuthology.orchestra.run.smithi090.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-21T23:37:38.647 INFO:teuthology.orchestra.run.smithi090.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-02-21T23:37:38.647 INFO:teuthology.orchestra.run.smithi090.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-02-21T23:37:38.647 INFO:teuthology.orchestra.run.smithi090.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-21T23:37:38.647 INFO:teuthology.orchestra.run.smithi090.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-02-21T23:37:38.647 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-02-21T23:37:38.647 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-02-21T23:37:38.647 INFO:teuthology.orchestra.run.smithi090.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-02-21T23:37:38.647 INFO:teuthology.orchestra.run.smithi090.stdout: python3-packaging noarch 16.8-9.el8 codeready-builder-for-rhel-8-x86_64-rpms 48 k 2024-02-21T23:37:38.647 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-21T23:37:38.647 INFO:teuthology.orchestra.run.smithi090.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-21T23:37:38.647 INFO:teuthology.orchestra.run.smithi090.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-02-21T23:37:38.647 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-02-21T23:37:38.648 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-02-21T23:37:38.648 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-02-21T23:37:38.648 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-02-21T23:37:38.648 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-02-21T23:37:38.648 INFO:teuthology.orchestra.run.smithi090.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-21T23:37:38.648 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-02-21T23:37:38.648 INFO:teuthology.orchestra.run.smithi090.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-21T23:37:38.648 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-21T23:37:38.648 INFO:teuthology.orchestra.run.smithi090.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-02-21T23:37:38.648 INFO:teuthology.orchestra.run.smithi090.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-21T23:37:38.648 INFO:teuthology.orchestra.run.smithi090.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-21T23:37:38.648 INFO:teuthology.orchestra.run.smithi090.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-21T23:37:38.648 INFO:teuthology.orchestra.run.smithi090.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-21T23:37:38.649 INFO:teuthology.orchestra.run.smithi090.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-21T23:37:38.649 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-21T23:37:38.649 INFO:teuthology.orchestra.run.smithi090.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-21T23:37:38.649 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-21T23:37:38.649 INFO:teuthology.orchestra.run.smithi090.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-02-21T23:37:38.649 INFO:teuthology.orchestra.run.smithi090.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-21T23:37:38.649 INFO:teuthology.orchestra.run.smithi090.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-02-21T23:37:38.649 INFO:teuthology.orchestra.run.smithi090.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-02-21T23:37:38.649 INFO:teuthology.orchestra.run.smithi090.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2024-02-21T23:37:38.650 INFO:teuthology.orchestra.run.smithi090.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-02-21T23:37:38.650 INFO:teuthology.orchestra.run.smithi090.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2024-02-21T23:37:38.650 INFO:teuthology.orchestra.run.smithi090.stdout:Enabling module streams: 2024-02-21T23:37:38.650 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql 10 2024-02-21T23:37:38.650 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:37:38.650 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-21T23:37:38.650 INFO:teuthology.orchestra.run.smithi090.stdout:=================================================================================================================================== 2024-02-21T23:37:38.650 INFO:teuthology.orchestra.run.smithi090.stdout:Install 130 Packages 2024-02-21T23:37:38.650 INFO:teuthology.orchestra.run.smithi090.stdout:Upgrade 16 Packages 2024-02-21T23:37:38.650 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:37:38.651 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 212 M 2024-02-21T23:37:38.651 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-21T23:37:38.907 INFO:teuthology.orchestra.run.smithi090.stdout:(1/146): ceph-18.2.1-638.gc1dc18f4.el8.x86_64.r 48 kB/s | 6.4 kB 00:00 2024-02-21T23:37:39.116 INFO:teuthology.orchestra.run.smithi090.stdout:(2/146): ceph-fuse-18.2.1-638.gc1dc18f4.el8.x86 4.0 MB/s | 853 kB 00:00 2024-02-21T23:37:39.183 INFO:teuthology.orchestra.run.smithi090.stdout:(3/146): ceph-immutable-object-cache-18.2.1-638 2.4 MB/s | 165 kB 00:00 2024-02-21T23:37:39.266 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-common-2:18.2.1-638.gc1dc18f4.el8.x86_64 105/162 2024-02-21T23:37:39.283 INFO:teuthology.orchestra.run.smithi090.stdout:(4/146): ceph-base-18.2.1-638.gc1dc18f4.el8.x86 11 MB/s | 5.4 MB 00:00 2024-02-21T23:37:39.417 INFO:teuthology.orchestra.run.smithi090.stdout:(5/146): ceph-mds-18.2.1-638.gc1dc18f4.el8.x86_ 9.8 MB/s | 2.3 MB 00:00 2024-02-21T23:37:39.459 INFO:teuthology.orchestra.run.smithi090.stdout:(6/146): ceph-mgr-18.2.1-638.gc1dc18f4.el8.x86_ 8.9 MB/s | 1.6 MB 00:00 2024-02-21T23:37:39.884 INFO:teuthology.orchestra.run.smithi090.stdout:(7/146): ceph-mon-18.2.1-638.gc1dc18f4.el8.x86_ 9.9 MB/s | 4.6 MB 00:00 2024-02-21T23:37:39.922 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-common-2:18.2.1-638.gc1dc18f4.el8.x86_64 105/162 2024-02-21T23:37:40.512 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-common-2:18.2.1-638.gc1dc18f4.el8.x86_64 105/162 2024-02-21T23:37:40.837 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-base-2:18.2.1-638.gc1dc18f4.el8.x86_64 106/162 2024-02-21T23:37:40.923 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-base-2:18.2.1-638.gc1dc18f4.el8.x86_64 106/162 2024-02-21T23:37:41.075 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-selinux-2:18.2.1-638.gc1dc18f4.el8.x86_64 107/162 2024-02-21T23:37:41.202 INFO:teuthology.orchestra.run.smithi090.stdout:(8/146): ceph-osd-18.2.1-638.gc1dc18f4.el8.x86_ 10 MB/s | 18 MB 00:01 2024-02-21T23:37:41.268 INFO:teuthology.orchestra.run.smithi090.stdout:(9/146): ceph-selinux-18.2.1-638.gc1dc18f4.el8. 366 kB/s | 24 kB 00:00 2024-02-21T23:37:41.653 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-common-2:18.2.1-638.gc1dc18f4.el8.x86_64 105/162 2024-02-21T23:37:41.710 INFO:teuthology.orchestra.run.smithi090.stdout:(10/146): ceph-radosgw-18.2.1-638.gc1dc18f4.el8 7.2 MB/s | 13 MB 00:01 2024-02-21T23:37:41.802 INFO:teuthology.orchestra.run.smithi090.stdout:(11/146): cephfs-mirror-18.2.1-638.gc1dc18f4.el 2.8 MB/s | 259 kB 00:00 2024-02-21T23:37:41.869 INFO:teuthology.orchestra.run.smithi090.stdout:(12/146): libcephfs-devel-18.2.1-638.gc1dc18f4. 464 kB/s | 31 kB 00:00 2024-02-21T23:37:41.896 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-common-2:18.2.1-638.gc1dc18f4.el8.x86_64 105/162 2024-02-21T23:37:42.036 INFO:teuthology.orchestra.run.smithi090.stdout:(13/146): libcephfs2-18.2.1-638.gc1dc18f4.el8.x 4.5 MB/s | 775 kB 00:00 2024-02-21T23:37:42.112 INFO:teuthology.orchestra.run.smithi090.stdout:(14/146): libcephsqlite-18.2.1-638.gc1dc18f4.el 1.9 MB/s | 142 kB 00:00 2024-02-21T23:37:42.195 INFO:teuthology.orchestra.run.smithi090.stdout:(15/146): librados-devel-18.2.1-638.gc1dc18f4.e 1.6 MB/s | 139 kB 00:00 2024-02-21T23:37:42.296 INFO:teuthology.orchestra.run.smithi090.stdout:(16/146): libradosstriper1-18.2.1-638.gc1dc18f4 4.6 MB/s | 473 kB 00:00 2024-02-21T23:37:42.473 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-common-2:18.2.1-638.gc1dc18f4.el8.x86_64 105/162 2024-02-21T23:37:42.745 INFO:teuthology.orchestra.run.smithi090.stdout:(17/146): ceph-common-18.2.1-638.gc1dc18f4.el8. 5.5 MB/s | 22 MB 00:03 2024-02-21T23:37:42.790 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-base-2:18.2.1-638.gc1dc18f4.el8.x86_64 106/162 2024-02-21T23:37:42.830 INFO:teuthology.orchestra.run.smithi090.stdout:(18/146): python3-ceph-argparse-18.2.1-638.gc1d 529 kB/s | 45 kB 00:00 2024-02-21T23:37:42.860 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-base-2:18.2.1-638.gc1dc18f4.el8.x86_64 106/162 2024-02-21T23:37:42.922 INFO:teuthology.orchestra.run.smithi090.stdout:(19/146): python3-ceph-common-18.2.1-638.gc1dc1 1.3 MB/s | 125 kB 00:00 2024-02-21T23:37:43.014 INFO:teuthology.orchestra.run.smithi090.stdout:(20/146): python3-cephfs-18.2.1-638.gc1dc18f4.e 2.1 MB/s | 196 kB 00:00 2024-02-21T23:37:43.045 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-selinux-2:18.2.1-638.gc1dc18f4.el8.x86_64 107/162 2024-02-21T23:37:43.182 INFO:teuthology.orchestra.run.smithi090.stdout:(21/146): librgw2-18.2.1-638.gc1dc18f4.el8.x86_ 7.4 MB/s | 6.5 MB 00:00 2024-02-21T23:37:43.207 INFO:teuthology.orchestra.run.smithi090.stdout:(22/146): python3-rados-18.2.1-638.gc1dc18f4.el 1.9 MB/s | 371 kB 00:00 2024-02-21T23:37:43.249 INFO:teuthology.orchestra.run.smithi090.stdout:(23/146): python3-rbd-18.2.1-638.gc1dc18f4.el8. 5.3 MB/s | 364 kB 00:00 2024-02-21T23:37:43.283 INFO:teuthology.orchestra.run.smithi090.stdout:(24/146): python3-rgw-18.2.1-638.gc1dc18f4.el8. 1.4 MB/s | 106 kB 00:00 2024-02-21T23:37:43.308 INFO:teuthology.orchestra.run.smithi090.stdout:(25/146): rbd-fuse-18.2.1-638.gc1dc18f4.el8.x86 1.6 MB/s | 95 kB 00:00 2024-02-21T23:37:43.392 INFO:teuthology.orchestra.run.smithi090.stdout:(26/146): rbd-nbd-18.2.1-638.gc1dc18f4.el8.x86_ 2.1 MB/s | 182 kB 00:00 2024-02-21T23:37:43.459 INFO:teuthology.orchestra.run.smithi090.stdout:(27/146): ceph-grafana-dashboards-18.2.1-638.gc 345 kB/s | 23 kB 00:00 2024-02-21T23:37:43.543 INFO:teuthology.orchestra.run.smithi090.stdout:(28/146): ceph-mgr-cephadm-18.2.1-638.gc1dc18f4 1.6 MB/s | 137 kB 00:00 2024-02-21T23:37:43.618 INFO:teuthology.orchestra.run.smithi090.stdout:(29/146): rbd-mirror-18.2.1-638.gc1dc18f4.el8.x 10 MB/s | 3.5 MB 00:00 2024-02-21T23:37:44.118 INFO:teuthology.orchestra.run.smithi090.stdout:(30/146): ceph-mgr-dashboard-18.2.1-638.gc1dc18 3.0 MB/s | 1.7 MB 00:00 2024-02-21T23:37:44.229 INFO:teuthology.orchestra.run.smithi090.stdout:(31/146): ceph-mgr-modules-core-18.2.1-638.gc1d 2.2 MB/s | 244 kB 00:00 2024-02-21T23:37:44.336 INFO:teuthology.orchestra.run.smithi090.stdout:(32/146): ceph-mgr-diskprediction-local-18.2.1- 10 MB/s | 7.4 MB 00:00 2024-02-21T23:37:44.361 INFO:teuthology.orchestra.run.smithi090.stdout:(33/146): ceph-mgr-rook-18.2.1-638.gc1dc18f4.el 395 kB/s | 52 kB 00:00 2024-02-21T23:37:44.411 INFO:teuthology.orchestra.run.smithi090.stdout:(34/146): ceph-prometheus-alerts-18.2.1-638.gc1 196 kB/s | 15 kB 00:00 2024-02-21T23:37:44.437 INFO:teuthology.orchestra.run.smithi090.stdout:(35/146): ceph-volume-18.2.1-638.gc1dc18f4.el8. 3.4 MB/s | 263 kB 00:00 2024-02-21T23:37:44.479 INFO:teuthology.orchestra.run.smithi090.stdout:(36/146): cephadm-18.2.1-638.gc1dc18f4.el8.noar 2.5 MB/s | 172 kB 00:00 2024-02-21T23:37:44.504 INFO:teuthology.orchestra.run.smithi090.stdout:(37/146): cephfs-top-18.2.1-638.gc1dc18f4.el8.n 267 kB/s | 18 kB 00:00 2024-02-21T23:37:45.055 INFO:teuthology.orchestra.run.smithi090.stdout:(38/146): ceph-test-18.2.1-638.gc1dc18f4.el8.x8 12 MB/s | 47 MB 00:03 2024-02-21T23:37:45.305 INFO:teuthology.orchestra.run.smithi090.stdout:(39/146): liboath-2.6.2-3.el8.x86_64.rpm 73 kB/s | 59 kB 00:00 2024-02-21T23:37:45.372 INFO:teuthology.orchestra.run.smithi090.stdout:(40/146): libunwind-1.3.1-3.el8.x86_64.rpm 243 kB/s | 75 kB 00:00 2024-02-21T23:37:45.414 INFO:teuthology.orchestra.run.smithi090.stdout:(41/146): gperftools-libs-2.7-9.el8.x86_64.rpm 327 kB/s | 306 kB 00:00 2024-02-21T23:37:45.472 INFO:teuthology.orchestra.run.smithi090.stdout:(42/146): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 435 kB/s | 44 kB 00:00 2024-02-21T23:37:45.514 INFO:teuthology.orchestra.run.smithi090.stdout:(43/146): python3-beautifulsoup4-4.6.3-2.el8.1. 1.8 MB/s | 185 kB 00:00 2024-02-21T23:37:45.556 INFO:teuthology.orchestra.run.smithi090.stdout:(44/146): python3-asyncssh-2.7.0-2.el8.noarch.r 1.8 MB/s | 476 kB 00:00 2024-02-21T23:37:45.583 INFO:teuthology.orchestra.run.smithi090.stdout:(45/146): python3-cachetools-3.1.1-4.el8.noarch 299 kB/s | 33 kB 00:00 2024-02-21T23:37:45.608 INFO:teuthology.orchestra.run.smithi090.stdout:(46/146): python3-certifi-2018.10.15-7.el8.noar 175 kB/s | 16 kB 00:00 2024-02-21T23:37:45.650 INFO:teuthology.orchestra.run.smithi090.stdout:(47/146): python3-cheroot-8.5.2-1.el8.noarch.rp 1.8 MB/s | 173 kB 00:00 2024-02-21T23:37:45.726 INFO:teuthology.orchestra.run.smithi090.stdout:(48/146): python3-google-auth-1.1.1-10.el8.noar 694 kB/s | 81 kB 00:00 2024-02-21T23:37:45.751 INFO:teuthology.orchestra.run.smithi090.stdout:(49/146): python3-jaraco-6.2-6.el8.noarch.rpm 111 kB/s | 11 kB 00:00 2024-02-21T23:37:45.801 INFO:teuthology.orchestra.run.smithi090.stdout:(50/146): python3-jaraco-functools-2.0-4.el8.no 233 kB/s | 18 kB 00:00 2024-02-21T23:37:45.835 INFO:teuthology.orchestra.run.smithi090.stdout:(51/146): python3-cherrypy-18.4.0-1.el8.noarch. 1.5 MB/s | 384 kB 00:00 2024-02-21T23:37:45.893 INFO:teuthology.orchestra.run.smithi090.stdout:(52/146): python3-logutils-0.3.5-11.el8.noarch. 531 kB/s | 49 kB 00:00 2024-02-21T23:37:45.927 INFO:teuthology.orchestra.run.smithi090.stdout:(53/146): python3-more-itertools-7.2.0-3.el8.no 638 kB/s | 59 kB 00:00 2024-02-21T23:37:45.961 INFO:teuthology.orchestra.run.smithi090.stdout:(54/146): python3-kubernetes-11.0.0-6.el8.noarc 5.6 MB/s | 1.2 MB 00:00 2024-02-21T23:37:45.986 INFO:teuthology.orchestra.run.smithi090.stdout:(55/146): python3-natsort-7.1.1-2.el8.noarch.rp 652 kB/s | 60 kB 00:00 2024-02-21T23:37:46.068 INFO:teuthology.orchestra.run.smithi090.stdout:(56/146): python3-portend-2.6-1.el8.noarch.rpm 146 kB/s | 16 kB 00:00 2024-02-21T23:37:46.103 INFO:teuthology.orchestra.run.smithi090.stdout:(57/146): python3-pecan-1.3.2-9.el8.noarch.rpm 1.6 MB/s | 283 kB 00:00 2024-02-21T23:37:46.129 INFO:teuthology.orchestra.run.smithi090.stdout:(58/146): python3-repoze-lru-0.7-6.el8.noarch.r 236 kB/s | 34 kB 00:00 2024-02-21T23:37:46.154 INFO:teuthology.orchestra.run.smithi090.stdout:(59/146): python3-routes-2.4.1-12.el8.noarch.rp 2.2 MB/s | 196 kB 00:00 2024-02-21T23:37:46.196 INFO:teuthology.orchestra.run.smithi090.stdout:(60/146): python3-rsa-4.9-2.el8.noarch.rpm 695 kB/s | 64 kB 00:00 2024-02-21T23:37:46.222 INFO:teuthology.orchestra.run.smithi090.stdout:(61/146): python3-simplegeneric-0.8.1-17.el8.no 203 kB/s | 19 kB 00:00 2024-02-21T23:37:46.247 INFO:teuthology.orchestra.run.smithi090.stdout:(62/146): python3-singledispatch-3.4.0.3-18.el8 259 kB/s | 24 kB 00:00 2024-02-21T23:37:46.281 INFO:teuthology.orchestra.run.smithi090.stdout:(63/146): python3-tempora-1.14.1-5.el8.noarch.r 349 kB/s | 29 kB 00:00 2024-02-21T23:37:46.384 INFO:teuthology.orchestra.run.smithi090.stdout:(64/146): python3-trustme-0.6.0-4.el8.noarch.rp 170 kB/s | 27 kB 00:00 2024-02-21T23:37:46.448 INFO:teuthology.orchestra.run.smithi090.stdout:(65/146): python3-waitress-1.4.3-1.el8.noarch.r 1.2 MB/s | 255 kB 00:00 2024-02-21T23:37:46.473 INFO:teuthology.orchestra.run.smithi090.stdout:(66/146): python3-webob-1.8.5-1.el8.1.noarch.rp 1.3 MB/s | 251 kB 00:00 2024-02-21T23:37:46.498 INFO:teuthology.orchestra.run.smithi090.stdout:(67/146): python3-websocket-client-0.56.0-5.el8 538 kB/s | 61 kB 00:00 2024-02-21T23:37:46.548 INFO:teuthology.orchestra.run.smithi090.stdout:(68/146): python3-zc-lockfile-2.0-2.el8.noarch. 306 kB/s | 23 kB 00:00 2024-02-21T23:37:46.574 INFO:teuthology.orchestra.run.smithi090.stdout:(69/146): python3-webtest-2.0.33-1.el8.noarch.r 851 kB/s | 85 kB 00:00 2024-02-21T23:37:46.599 INFO:teuthology.orchestra.run.smithi090.stdout:(70/146): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 70 kB 00:00 2024-02-21T23:37:46.709 INFO:teuthology.orchestra.run.smithi090.stdout:(71/146): libverto-devel-0.3.0-5.el8.x86_64.rpm 172 kB/s | 18 kB 00:00 2024-02-21T23:37:46.790 INFO:teuthology.orchestra.run.smithi090.stdout:(72/146): thrift-0.13.0-2.el8.x86_64.rpm 5.9 MB/s | 1.7 MB 00:00 2024-02-21T23:37:46.968 INFO:teuthology.orchestra.run.smithi090.stdout:(73/146): python3-packaging-16.8-9.el8.noarch.r 122 kB/s | 48 kB 00:00 2024-02-21T23:37:47.002 INFO:teuthology.orchestra.run.smithi090.stdout:(74/146): python3-oauthlib-2.1.0-1.el8.noarch.r 734 kB/s | 155 kB 00:00 2024-02-21T23:37:47.061 INFO:teuthology.orchestra.run.smithi090.stdout:(75/146): python3-jwt-1.6.1-2.el8.noarch.rpm 121 kB/s | 43 kB 00:00 2024-02-21T23:37:47.094 INFO:teuthology.orchestra.run.smithi090.stdout:(76/146): python3-pycparser-2.14-14.el8.noarch. 869 kB/s | 109 kB 00:00 2024-02-21T23:37:47.119 INFO:teuthology.orchestra.run.smithi090.stdout:(77/146): python3-requests-oauthlib-1.0.0-1.el8 364 kB/s | 43 kB 00:00 2024-02-21T23:37:47.186 INFO:teuthology.orchestra.run.smithi090.stdout:(78/146): libaio-devel-0.3.112-1.el8.x86_64.rpm 149 kB/s | 19 kB 00:00 2024-02-21T23:37:47.212 INFO:teuthology.orchestra.run.smithi090.stdout:(79/146): libzstd-devel-1.4.4-1.el8.x86_64.rpm 373 kB/s | 44 kB 00:00 2024-02-21T23:37:47.253 INFO:teuthology.orchestra.run.smithi090.stdout:(80/146): pcre2-devel-10.32-2.el8.x86_64.rpm 4.4 MB/s | 605 kB 00:00 2024-02-21T23:37:47.329 INFO:teuthology.orchestra.run.smithi090.stdout:(81/146): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.6 MB/s | 229 kB 00:00 2024-02-21T23:37:47.363 INFO:teuthology.orchestra.run.smithi090.stdout:(82/146): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.4 MB/s | 220 kB 00:00 2024-02-21T23:37:47.396 INFO:teuthology.orchestra.run.smithi090.stdout:(83/146): libselinux-devel-2.9-5.el8.x86_64.rpm 1.4 MB/s | 200 kB 00:00 2024-02-21T23:37:47.472 INFO:teuthology.orchestra.run.smithi090.stdout:(84/146): keyutils-libs-devel-1.5.10-9.el8.x86_ 338 kB/s | 48 kB 00:00 2024-02-21T23:37:47.521 INFO:teuthology.orchestra.run.smithi090.stdout:(85/146): libsepol-devel-2.9-3.el8.x86_64.rpm 550 kB/s | 87 kB 00:00 2024-02-21T23:37:47.556 INFO:teuthology.orchestra.run.smithi090.stdout:(86/146): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 363 kB/s | 58 kB 00:00 2024-02-21T23:37:47.606 INFO:teuthology.orchestra.run.smithi090.stdout:(87/146): libcom_err-devel-1.45.6-4.el8.x86_64. 288 kB/s | 39 kB 00:00 2024-02-21T23:37:47.640 INFO:teuthology.orchestra.run.smithi090.stdout:(88/146): fuse-2.9.7-17.el8.x86_64.rpm 703 kB/s | 83 kB 00:00 2024-02-21T23:37:47.682 INFO:teuthology.orchestra.run.smithi090.stdout:(89/146): libgfortran-8.5.0-20.el8.x86_64.rpm 5.0 MB/s | 645 kB 00:00 2024-02-21T23:37:47.715 INFO:teuthology.orchestra.run.smithi090.stdout:(90/146): libuuid-devel-2.32.1-43.el8.x86_64.rp 1.3 MB/s | 100 kB 00:00 2024-02-21T23:37:47.741 INFO:teuthology.orchestra.run.smithi090.stdout:(91/146): libquadmath-8.5.0-20.el8.x86_64.rpm 1.3 MB/s | 172 kB 00:00 2024-02-21T23:37:47.824 INFO:teuthology.orchestra.run.smithi090.stdout:(92/146): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.6 MB/s | 238 kB 00:00 2024-02-21T23:37:47.858 INFO:teuthology.orchestra.run.smithi090.stdout:(93/146): xfsprogs-devel-5.0.0-12.el8.x86_64.rp 620 kB/s | 73 kB 00:00 2024-02-21T23:37:47.917 INFO:teuthology.orchestra.run.smithi090.stdout:(94/146): python3-cryptography-3.2.1-6.el8.x86_ 2.7 MB/s | 559 kB 00:00 2024-02-21T23:37:47.950 INFO:teuthology.orchestra.run.smithi090.stdout:(95/146): NetworkManager-initscripts-updown-1.4 1.1 MB/s | 144 kB 00:00 2024-02-21T23:37:47.992 INFO:teuthology.orchestra.run.smithi090.stdout:(96/146): elfutils-libelf-devel-0.189-3.el8.x86 461 kB/s | 62 kB 00:00 2024-02-21T23:37:48.060 INFO:teuthology.orchestra.run.smithi090.stdout:(97/146): librabbitmq-0.9.0-4.el8.x86_64.rpm 427 kB/s | 47 kB 00:00 2024-02-21T23:37:48.102 INFO:teuthology.orchestra.run.smithi090.stdout:(98/146): krb5-devel-1.18.2-26.el8_9.x86_64.rpm 3.0 MB/s | 562 kB 00:00 2024-02-21T23:37:48.178 INFO:teuthology.orchestra.run.smithi090.stdout:(99/146): openssl-devel-1.1.1k-12.el8_9.x86_64. 13 MB/s | 2.3 MB 00:00 2024-02-21T23:37:48.203 INFO:teuthology.orchestra.run.smithi090.stdout:(100/146): python3-jsonpointer-1.10-11.el8.noar 141 kB/s | 20 kB 00:00 2024-02-21T23:37:48.253 INFO:teuthology.orchestra.run.smithi090.stdout:(101/146): python3-prettytable-0.7.2-14.el8.noa 292 kB/s | 44 kB 00:00 2024-02-21T23:37:48.304 INFO:teuthology.orchestra.run.smithi090.stdout:(102/146): python3-pyasn1-modules-0.3.7-6.el8.n 876 kB/s | 110 kB 00:00 2024-02-21T23:37:48.396 INFO:teuthology.orchestra.run.smithi090.stdout:(103/146): python3-jsonpatch-1.21-2.el8.noarch. 138 kB/s | 27 kB 00:00 2024-02-21T23:37:48.454 INFO:teuthology.orchestra.run.smithi090.stdout:(104/146): python3-pyasn1-0.3.7-6.el8.noarch.rp 629 kB/s | 126 kB 00:00 2024-02-21T23:37:48.488 INFO:teuthology.orchestra.run.smithi090.stdout:(105/146): python3-werkzeug-0.12.2-4.el8.noarch 2.4 MB/s | 457 kB 00:00 2024-02-21T23:37:48.531 INFO:teuthology.orchestra.run.smithi090.stdout:(106/146): bison-3.0.4-10.el8.x86_64.rpm 5.0 MB/s | 688 kB 00:00 2024-02-21T23:37:48.647 INFO:teuthology.orchestra.run.smithi090.stdout:(107/146): libtool-2.4.6-25.el8.x86_64.rpm 4.4 MB/s | 709 kB 00:00 2024-02-21T23:37:48.672 INFO:teuthology.orchestra.run.smithi090.stdout:(108/146): python3-markupsafe-0.23-19.el8.x86_6 273 kB/s | 39 kB 00:00 2024-02-21T23:37:48.706 INFO:teuthology.orchestra.run.smithi090.stdout:(109/146): flex-2.6.1-9.el8.x86_64.rpm 1.2 MB/s | 320 kB 00:00 2024-02-21T23:37:48.790 INFO:teuthology.orchestra.run.smithi090.stdout:(110/146): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.8 MB/s | 259 kB 00:00 2024-02-21T23:37:48.815 INFO:teuthology.orchestra.run.smithi090.stdout:(111/146): uuid-1.6.2-43.el8.x86_64.rpm 588 kB/s | 64 kB 00:00 2024-02-21T23:37:48.849 INFO:teuthology.orchestra.run.smithi090.stdout:(112/146): oniguruma-6.8.2-2.el8.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-02-21T23:37:48.916 INFO:teuthology.orchestra.run.smithi090.stdout:(113/146): python3-jinja2-2.10.1-3.el8.noarch.r 7.9 MB/s | 538 kB 00:00 2024-02-21T23:37:48.991 INFO:teuthology.orchestra.run.smithi090.stdout:(114/146): python3-pyOpenSSL-19.0.0-1.el8.noarc 516 kB/s | 103 kB 00:00 2024-02-21T23:37:49.133 INFO:teuthology.orchestra.run.smithi090.stdout:(115/146): python3-babel-2.5.1-7.el8.noarch.rpm 15 MB/s | 4.8 MB 00:00 2024-02-21T23:37:49.166 INFO:teuthology.orchestra.run.smithi090.stdout:(116/146): socat-1.7.4.1-1.el8.x86_64.rpm 1.8 MB/s | 323 kB 00:00 2024-02-21T23:37:49.242 INFO:teuthology.orchestra.run.smithi090.stdout:(117/146): python3-numpy-1.14.3-10.el8.x86_64.r 11 MB/s | 3.7 MB 00:00 2024-02-21T23:37:49.267 INFO:teuthology.orchestra.run.smithi090.stdout:(118/146): python3-numpy-f2py-1.14.3-10.el8.x86 1.6 MB/s | 225 kB 00:00 2024-02-21T23:37:49.359 INFO:teuthology.orchestra.run.smithi090.stdout:(119/146): librdkafka-0.11.4-3.el8.x86_64.rpm 1.8 MB/s | 354 kB 00:00 2024-02-21T23:37:49.401 INFO:teuthology.orchestra.run.smithi090.stdout:(120/146): python3-mako-1.0.6-14.el8.noarch.rpm 1.1 MB/s | 157 kB 00:00 2024-02-21T23:37:49.510 INFO:teuthology.orchestra.run.smithi090.stdout:(121/146): openblas-threads-0.3.15-6.el8.x86_64 19 MB/s | 5.2 MB 00:00 2024-02-21T23:37:49.577 INFO:teuthology.orchestra.run.smithi090.stdout:(122/146): automake-1.16.1-8.el8.noarch.rpm 4.0 MB/s | 714 kB 00:00 2024-02-21T23:37:49.698 INFO:teuthology.orchestra.run.smithi090.stdout:(123/146): openblas-0.3.15-6.el8.x86_64.rpm 15 MB/s | 5.0 MB 00:00 2024-02-21T23:37:49.728 INFO:teuthology.orchestra.run.smithi090.stdout:(124/146): libpq-13.11-1.el8.x86_64.rpm 906 kB/s | 198 kB 00:00 2024-02-21T23:37:49.762 INFO:teuthology.orchestra.run.smithi090.stdout:(125/146): jq-1.6-7.el8.x86_64.rpm 1.1 MB/s | 202 kB 00:00 2024-02-21T23:37:49.838 INFO:teuthology.orchestra.run.smithi090.stdout:(126/146): python3-pytz-2017.2-11.el8.noarch.rp 389 kB/s | 54 kB 00:00 2024-02-21T23:37:49.896 INFO:teuthology.orchestra.run.smithi090.stdout:(127/146): postgresql-10.23-3.module+el8.9.0+20 9.0 MB/s | 1.5 MB 00:00 2024-02-21T23:37:49.938 INFO:teuthology.orchestra.run.smithi090.stdout:(128/146): postgresql-contrib-10.23-3.module+el 4.5 MB/s | 811 kB 00:00 2024-02-21T23:37:50.089 INFO:teuthology.orchestra.run.smithi090.stdout:(129/146): postgresql-server-10.23-3.module+el8 20 MB/s | 5.1 MB 00:00 2024-02-21T23:37:50.487 INFO:teuthology.orchestra.run.smithi090.stdout:(130/146): python3-scipy-1.0.0-21.module+el8.9. 23 MB/s | 14 MB 00:00 2024-02-21T23:37:50.556 INFO:teuthology.orchestra.run.smithi090.stdout:(131/146): librados2-18.2.1-638.gc1dc18f4.el8.x 5.9 MB/s | 3.7 MB 00:00 2024-02-21T23:37:50.623 INFO:teuthology.orchestra.run.smithi090.stdout:(132/146): librbd1-18.2.1-638.gc1dc18f4.el8.x86 6.7 MB/s | 3.6 MB 00:00 2024-02-21T23:37:50.657 INFO:teuthology.orchestra.run.smithi090.stdout:(133/146): elfutils-libelf-0.189-3.el8.x86_64.r 1.3 MB/s | 232 kB 00:00 2024-02-21T23:37:50.699 INFO:teuthology.orchestra.run.smithi090.stdout:(134/146): elfutils-libs-0.189-3.el8.x86_64.rpm 2.1 MB/s | 303 kB 00:00 2024-02-21T23:37:50.774 INFO:teuthology.orchestra.run.smithi090.stdout:(135/146): libsmartcols-2.32.1-43.el8.x86_64.rp 1.5 MB/s | 179 kB 00:00 2024-02-21T23:37:50.808 INFO:teuthology.orchestra.run.smithi090.stdout:(136/146): libmount-2.32.1-43.el8.x86_64.rpm 1.3 MB/s | 237 kB 00:00 2024-02-21T23:37:50.863 INFO:teuthology.orchestra.run.smithi090.stdout:(137/146): libuuid-2.32.1-43.el8.x86_64.rpm 601 kB/s | 99 kB 00:00 2024-02-21T23:37:50.966 INFO:teuthology.orchestra.run.smithi090.stdout:(138/146): util-linux-2.32.1-43.el8.x86_64.rpm 13 MB/s | 2.5 MB 00:00 2024-02-21T23:37:50.992 INFO:teuthology.orchestra.run.smithi090.stdout:(139/146): util-linux-user-2.32.1-43.el8.x86_64 558 kB/s | 103 kB 00:00 2024-02-21T23:37:51.034 INFO:teuthology.orchestra.run.smithi090.stdout:(140/146): iproute-6.2.0-5.el8_9.x86_64.rpm 4.9 MB/s | 854 kB 00:00 2024-02-21T23:37:51.109 INFO:teuthology.orchestra.run.smithi090.stdout:(141/146): libfdisk-2.32.1-43.el8.x86_64.rpm 2.1 MB/s | 254 kB 00:00 2024-02-21T23:37:51.134 INFO:teuthology.orchestra.run.smithi090.stdout:(142/146): libblkid-2.32.1-43.el8.x86_64.rpm 1.3 MB/s | 220 kB 00:00 2024-02-21T23:37:51.176 INFO:teuthology.orchestra.run.smithi090.stdout:(143/146): openssl-1.1.1k-12.el8_9.x86_64.rpm 4.9 MB/s | 711 kB 00:00 2024-02-21T23:37:51.260 INFO:teuthology.orchestra.run.smithi090.stdout:(144/146): openssl-libs-1.1.1k-12.el8_9.x86_64. 9.8 MB/s | 1.5 MB 00:00 2024-02-21T23:37:51.294 INFO:teuthology.orchestra.run.smithi090.stdout:(145/146): xfsdump-3.1.8-7.el8_9.x86_64.rpm 2.1 MB/s | 333 kB 00:00 2024-02-21T23:37:51.352 INFO:teuthology.orchestra.run.smithi090.stdout:(146/146): xfsprogs-5.0.0-12.el8.x86_64.rpm 6.1 MB/s | 1.1 MB 00:00 2024-02-21T23:37:51.355 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-21T23:37:51.355 INFO:teuthology.orchestra.run.smithi090.stdout:Total 17 MB/s | 212 MB 00:12 2024-02-21T23:37:52.412 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-21T23:37:52.804 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-21T23:37:52.805 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-21T23:37:53.981 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-21T23:37:53.991 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-21T23:37:59.891 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-21T23:38:00.025 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/1 2024-02-21T23:38:00.306 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-02-21T23:38:00.520 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-02-21T23:38:00.612 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-selinux-2:18.2.1-638.gc1dc18f4.el8.x86_64 107/162 2024-02-21T23:38:00.612 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /sys 2024-02-21T23:38:00.612 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /proc 2024-02-21T23:38:00.612 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /dev 2024-02-21T23:38:00.612 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /run 2024-02-21T23:38:00.612 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /mnt 2024-02-21T23:38:00.612 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /var/tmp 2024-02-21T23:38:00.612 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /home 2024-02-21T23:38:00.612 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /tmp 2024-02-21T23:38:00.612 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /dev 2024-02-21T23:38:00.613 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-21T23:38:00.826 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/162 2024-02-21T23:38:00.972 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-volume-2:18.2.1-638.gc1dc18f4.el8.noarch 108/162 2024-02-21T23:38:01.034 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 3/162 2024-02-21T23:38:01.107 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 3/162 2024-02-21T23:38:01.418 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 4/162 2024-02-21T23:38:01.493 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 4/162 2024-02-21T23:38:01.698 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/162 2024-02-21T23:38:02.001 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 6/162 2024-02-21T23:38:02.542 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-volume-2:18.2.1-638.gc1dc18f4.el8.noarch 108/162 2024-02-21T23:38:02.685 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 7/162 2024-02-21T23:38:02.699 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-osd-2:18.2.1-638.gc1dc18f4.el8.x86_64 109/162 2024-02-21T23:38:02.769 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-selinux-2:18.2.1-638.gc1dc18f4.el8.x86_64 107/162 2024-02-21T23:38:02.769 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /sys 2024-02-21T23:38:02.769 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /proc 2024-02-21T23:38:02.769 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /dev 2024-02-21T23:38:02.769 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /run 2024-02-21T23:38:02.769 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /mnt 2024-02-21T23:38:02.769 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /var/tmp 2024-02-21T23:38:02.769 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /home 2024-02-21T23:38:02.769 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /tmp 2024-02-21T23:38:02.769 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /dev 2024-02-21T23:38:02.769 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-21T23:38:02.821 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 7/162 2024-02-21T23:38:02.919 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-osd-2:18.2.1-638.gc1dc18f4.el8.x86_64 109/162 2024-02-21T23:38:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-02-21T23:38:03.097 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mds-2:18.2.1-638.gc1dc18f4.el8.x86_64 110/162 2024-02-21T23:38:03.184 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-volume-2:18.2.1-638.gc1dc18f4.el8.noarch 108/162 2024-02-21T23:38:03.451 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-02-21T23:38:03.553 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mds-2:18.2.1-638.gc1dc18f4.el8.x86_64 110/162 2024-02-21T23:38:03.763 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mon-2:18.2.1-638.gc1dc18f4.el8.x86_64 111/162 2024-02-21T23:38:03.781 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mon-2:18.2.1-638.gc1dc18f4.el8.x86_64 111/162 2024-02-21T23:38:03.868 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : librados2-2:18.2.1-638.gc1dc18f4.el8.x86_64 9/162 2024-02-21T23:38:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados2-2:18.2.1-638.gc1dc18f4.el8.x86_64 9/162 2024-02-21T23:38:04.459 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : librbd1-2:18.2.1-638.gc1dc18f4.el8.x86_64 10/162 2024-02-21T23:38:04.604 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librbd1-2:18.2.1-638.gc1dc18f4.el8.x86_64 10/162 2024-02-21T23:38:04.754 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-volume-2:18.2.1-638.gc1dc18f4.el8.noarch 108/162 2024-02-21T23:38:04.961 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-osd-2:18.2.1-638.gc1dc18f4.el8.x86_64 109/162 2024-02-21T23:38:04.984 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephfs2-2:18.2.1-638.gc1dc18f4.el8.x86_64 11/162 2024-02-21T23:38:05.067 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephfs2-2:18.2.1-638.gc1dc18f4.el8.x86_64 11/162 2024-02-21T23:38:05.182 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-osd-2:18.2.1-638.gc1dc18f4.el8.x86_64 109/162 2024-02-21T23:38:05.316 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mds-2:18.2.1-638.gc1dc18f4.el8.x86_64 110/162 2024-02-21T23:38:05.509 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: cephadm-2:18.2.1-638.gc1dc18f4.el8.noarch 112/162 2024-02-21T23:38:05.562 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rados-2:18.2.1-638.gc1dc18f4.el8.x86_64 12/162 2024-02-21T23:38:05.660 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : cephadm-2:18.2.1-638.gc1dc18f4.el8.noarch 112/162 2024-02-21T23:38:05.774 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mds-2:18.2.1-638.gc1dc18f4.el8.x86_64 110/162 2024-02-21T23:38:05.777 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-prometheus-alerts-2:18.2.1-638.gc1dc18f4.el 113/162 2024-02-21T23:38:05.919 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mon-2:18.2.1-638.gc1dc18f4.el8.x86_64 111/162 2024-02-21T23:38:05.940 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mon-2:18.2.1-638.gc1dc18f4.el8.x86_64 111/162 2024-02-21T23:38:05.971 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-grafana-dashboards-2:18.2.1-638.gc1dc18f4.e 114/162 2024-02-21T23:38:06.062 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-02-21T23:38:06.151 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-02-21T23:38:06.163 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-cephadm-2:18.2.1-638.gc1dc18f4.el8.noar 115/162 2024-02-21T23:38:06.382 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.1-638.gc1dc18f4.el8.noar 115/162 2024-02-21T23:38:06.508 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libpq-13.11-1.el8.x86_64 14/162 2024-02-21T23:38:06.699 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-dashboard-2:18.2.1-638.gc1dc18f4.el8.no 116/162 2024-02-21T23:38:06.802 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 15/162 2024-02-21T23:38:07.006 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 16/162 2024-02-21T23:38:07.576 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 17/162 2024-02-21T23:38:07.672 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.1-638.gc1dc18f4.el8.no 116/162 2024-02-21T23:38:07.819 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: cephadm-2:18.2.1-638.gc1dc18f4.el8.noarch 112/162 2024-02-21T23:38:07.836 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.1-638.gc1dc 117/162 2024-02-21T23:38:07.929 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.1-638.gc1dc 117/162 2024-02-21T23:38:07.988 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : cephadm-2:18.2.1-638.gc1dc18f4.el8.noarch 112/162 2024-02-21T23:38:08.142 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : postgresql-10.23-3.module+el8.9.0+20831+78489779 18/162 2024-02-21T23:38:08.146 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-prometheus-alerts-2:18.2.1-638.gc1dc18f4.el 113/162 2024-02-21T23:38:08.311 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-modules-core-2:18.2.1-638.gc1dc18f4.el8 118/162 2024-02-21T23:38:08.326 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-grafana-dashboards-2:18.2.1-638.gc1dc18f4.e 114/162 2024-02-21T23:38:08.499 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-2:18.2.1-638.gc1dc18f4.el8.x86_64 119/162 2024-02-21T23:38:08.538 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-2:18.2.1-638.gc1dc18f4.el8.x86_64 119/162 2024-02-21T23:38:08.557 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-cephadm-2:18.2.1-638.gc1dc18f4.el8.noar 115/162 2024-02-21T23:38:08.687 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-rook-2:18.2.1-638.gc1dc18f4.el8.noarch 120/162 2024-02-21T23:38:08.699 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.1-638.gc1dc18f4.el8.noarch 120/162 2024-02-21T23:38:08.781 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.1-638.gc1dc18f4.el8.noar 115/162 2024-02-21T23:38:08.801 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephsqlite-2:18.2.1-638.gc1dc18f4.el8.x86_64 19/162 2024-02-21T23:38:08.907 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephsqlite-2:18.2.1-638.gc1dc18f4.el8.x86_64 19/162 2024-02-21T23:38:09.060 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libradosstriper1-2:18.2.1-638.gc1dc18f4.el8.x86_ 20/162 2024-02-21T23:38:09.126 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-dashboard-2:18.2.1-638.gc1dc18f4.el8.no 116/162 2024-02-21T23:38:09.138 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libradosstriper1-2:18.2.1-638.gc1dc18f4.el8.x86_ 20/162 2024-02-21T23:38:09.432 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 21/162 2024-02-21T23:38:09.691 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 22/162 2024-02-21T23:38:09.986 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-2:18.2.1-638.gc1dc18f4.el8.x86_64 121/162 2024-02-21T23:38:10.040 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 23/162 2024-02-21T23:38:10.092 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.1-638.gc1dc18f4.el8.no 116/162 2024-02-21T23:38:10.207 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-radosgw-2:18.2.1-638.gc1dc18f4.el8.x86_64 122/162 2024-02-21T23:38:10.209 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 24/162 2024-02-21T23:38:10.241 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-radosgw-2:18.2.1-638.gc1dc18f4.el8.x86_64 122/162 2024-02-21T23:38:10.325 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.1-638.gc1dc 117/162 2024-02-21T23:38:10.350 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 24/162 2024-02-21T23:38:10.418 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.1-638.gc1dc 117/162 2024-02-21T23:38:10.421 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-immutable-object-cache-2:18.2.1-638.gc1dc18 123/162 2024-02-21T23:38:10.464 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.1-638.gc1dc18 123/162 2024-02-21T23:38:10.618 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 25/162 2024-02-21T23:38:10.671 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : cephfs-mirror-2:18.2.1-638.gc1dc18f4.el8.x86_64 124/162 2024-02-21T23:38:10.797 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-modules-core-2:18.2.1-638.gc1dc18f4.el8 118/162 2024-02-21T23:38:10.985 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-2:18.2.1-638.gc1dc18f4.el8.x86_64 119/162 2024-02-21T23:38:11.018 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: cephfs-mirror-2:18.2.1-638.gc1dc18f4.el8.x86_64 124/162 2024-02-21T23:38:11.024 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-2:18.2.1-638.gc1dc18f4.el8.x86_64 119/162 2024-02-21T23:38:11.205 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-mirror-2:18.2.1-638.gc1dc18f4.el8.x86_64 125/162 2024-02-21T23:38:11.206 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-rook-2:18.2.1-638.gc1dc18f4.el8.noarch 120/162 2024-02-21T23:38:11.218 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.1-638.gc1dc18f4.el8.noarch 120/162 2024-02-21T23:38:11.243 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 25/162 2024-02-21T23:38:11.401 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 26/162 2024-02-21T23:38:12.038 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 26/162 2024-02-21T23:38:12.284 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openblas-0.3.15-6.el8.x86_64 27/162 2024-02-21T23:38:12.478 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-2:18.2.1-638.gc1dc18f4.el8.x86_64 121/162 2024-02-21T23:38:12.734 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-radosgw-2:18.2.1-638.gc1dc18f4.el8.x86_64 122/162 2024-02-21T23:38:12.769 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-radosgw-2:18.2.1-638.gc1dc18f4.el8.x86_64 122/162 2024-02-21T23:38:12.969 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 27/162 2024-02-21T23:38:12.973 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-immutable-object-cache-2:18.2.1-638.gc1dc18 123/162 2024-02-21T23:38:13.017 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.1-638.gc1dc18 123/162 2024-02-21T23:38:13.223 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : cephfs-mirror-2:18.2.1-638.gc1dc18f4.el8.x86_64 124/162 2024-02-21T23:38:13.241 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 28/162 2024-02-21T23:38:13.481 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 29/162 2024-02-21T23:38:13.571 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: cephfs-mirror-2:18.2.1-638.gc1dc18f4.el8.x86_64 124/162 2024-02-21T23:38:13.731 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-mirror-2:18.2.1-638.gc1dc18f4.el8.x86_64 125/162 2024-02-21T23:38:13.744 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 30/162 2024-02-21T23:38:13.928 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ceph-common-2:18.2.1-638.gc1dc18f4.el8.x 31/162 2024-02-21T23:38:14.104 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ceph-argparse-2:18.2.1-638.gc1dc18f4.el8 32/162 2024-02-21T23:38:14.372 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cephfs-2:18.2.1-638.gc1dc18f4.el8.x86_64 33/162 2024-02-21T23:38:14.559 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 34/162 2024-02-21T23:38:15.643 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: rbd-mirror-2:18.2.1-638.gc1dc18f4.el8.x86_64 125/162 2024-02-21T23:38:16.145 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-test-2:18.2.1-638.gc1dc18f4.el8.x86_64 126/162 2024-02-21T23:38:16.300 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 34/162 2024-02-21T23:38:16.472 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : postgresql-contrib-10.23-3.module+el8.9.0+20831+ 127/162 2024-02-21T23:38:16.636 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libtool-2.4.6-25.el8.x86_64 128/162 2024-02-21T23:38:16.695 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 35/162 2024-02-21T23:38:17.325 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 128/162 2024-02-21T23:38:17.641 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : openssl-devel-1:1.1.1k-12.el8_9.x86_64 129/162 2024-02-21T23:38:17.664 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 36/162 2024-02-21T23:38:17.903 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 130/162 2024-02-21T23:38:17.997 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 37/162 2024-02-21T23:38:18.126 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-fuse-2:18.2.1-638.gc1dc18f4.el8.x86_64 131/162 2024-02-21T23:38:18.158 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-mirror-2:18.2.1-638.gc1dc18f4.el8.x86_64 125/162 2024-02-21T23:38:18.217 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 38/162 2024-02-21T23:38:18.339 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 132/162 2024-02-21T23:38:18.470 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rsa-4.9-2.el8.noarch 39/162 2024-02-21T23:38:18.515 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 133/162 2024-02-21T23:38:18.597 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-test-2:18.2.1-638.gc1dc18f4.el8.x86_64 126/162 2024-02-21T23:38:18.694 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 40/162 2024-02-21T23:38:18.696 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcephfs-devel-2:18.2.1-638.gc1dc18f4.el8.x86_6 134/162 2024-02-21T23:38:18.931 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rbd-2:18.2.1-638.gc1dc18f4.el8.x86_64 41/162 2024-02-21T23:38:18.940 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : postgresql-contrib-10.23-3.module+el8.9.0+20831+ 127/162 2024-02-21T23:38:19.159 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librados-devel-2:18.2.1-638.gc1dc18f4.el8.x86_64 42/162 2024-02-21T23:38:19.163 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libtool-2.4.6-25.el8.x86_64 128/162 2024-02-21T23:38:19.334 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 43/162 2024-02-21T23:38:19.415 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 43/162 2024-02-21T23:38:19.675 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 44/162 2024-02-21T23:38:19.749 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 44/162 2024-02-21T23:38:19.847 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 128/162 2024-02-21T23:38:20.012 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 45/162 2024-02-21T23:38:20.176 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : openssl-devel-1:1.1.1k-12.el8_9.x86_64 129/162 2024-02-21T23:38:20.244 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 46/162 2024-02-21T23:38:20.442 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-02-21T23:38:20.470 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 130/162 2024-02-21T23:38:20.694 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-fuse-2:18.2.1-638.gc1dc18f4.el8.x86_64 131/162 2024-02-21T23:38:20.930 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: postgresql-server-10.23-3.module+el8.9.0+20831+7 135/162 2024-02-21T23:38:20.932 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 132/162 2024-02-21T23:38:20.959 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-02-21T23:38:21.151 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 133/162 2024-02-21T23:38:21.211 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : postgresql-server-10.23-3.module+el8.9.0+20831+7 135/162 2024-02-21T23:38:21.250 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: postgresql-server-10.23-3.module+el8.9.0+20831+7 135/162 2024-02-21T23:38:21.309 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 48/162 2024-02-21T23:38:21.323 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephfs-devel-2:18.2.1-638.gc1dc18f4.el8.x86_6 134/162 2024-02-21T23:38:21.404 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 48/162 2024-02-21T23:38:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : automake-1.16.1-8.el8.noarch 49/162 2024-02-21T23:38:21.647 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 136/162 2024-02-21T23:38:21.801 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : uuid-1.6.2-43.el8.x86_64 50/162 2024-02-21T23:38:21.887 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 50/162 2024-02-21T23:38:21.976 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : iproute-6.2.0-5.el8_9.x86_64 137/162 2024-02-21T23:38:22.084 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 51/162 2024-02-21T23:38:22.154 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 51/162 2024-02-21T23:38:22.189 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : xfsdump-3.1.8-7.el8_9.x86_64 138/162 2024-02-21T23:38:22.342 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : jq-1.6-7.el8.x86_64 52/162 2024-02-21T23:38:22.377 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : cephfs-top-2:18.2.1-638.gc1dc18f4.el8.noarch 139/162 2024-02-21T23:38:22.557 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 53/162 2024-02-21T23:38:22.643 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-fuse-2:18.2.1-638.gc1dc18f4.el8.x86_64 140/162 2024-02-21T23:38:22.767 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 54/162 2024-02-21T23:38:22.929 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-nbd-2:18.2.1-638.gc1dc18f4.el8.x86_64 141/162 2024-02-21T23:38:22.977 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 55/162 2024-02-21T23:38:23.195 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : openssl-1:1.1.1k-12.el8_9.x86_64 142/162 2024-02-21T23:38:23.222 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : fuse-2.9.7-17.el8.x86_64 56/162 2024-02-21T23:38:23.377 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : NetworkManager-initscripts-updown-1:1.40.16-13.e 143/162 2024-02-21T23:38:23.386 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-13.e 143/162 2024-02-21T23:38:23.433 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 57/162 2024-02-21T23:38:23.555 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 144/162 2024-02-21T23:38:23.588 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: postgresql-server-10.23-3.module+el8.9.0+20831+7 135/162 2024-02-21T23:38:23.631 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 58/162 2024-02-21T23:38:23.710 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : flex-2.6.1-9.el8.x86_64 145/162 2024-02-21T23:38:23.787 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 145/162 2024-02-21T23:38:23.905 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : postgresql-server-10.23-3.module+el8.9.0+20831+7 135/162 2024-02-21T23:38:23.912 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 59/162 2024-02-21T23:38:23.943 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: postgresql-server-10.23-3.module+el8.9.0+20831+7 135/162 2024-02-21T23:38:23.969 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : bison-3.0.4-10.el8.x86_64 146/162 2024-02-21T23:38:23.981 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 146/162 2024-02-21T23:38:24.104 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 60/162 2024-02-21T23:38:24.141 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : elfutils-libs-0.186-1.el8.x86_64 147/162 2024-02-21T23:38:24.353 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 148/162 2024-02-21T23:38:24.412 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 136/162 2024-02-21T23:38:24.446 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 61/162 2024-02-21T23:38:24.522 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 149/162 2024-02-21T23:38:24.522 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-02-21T23:38:24.746 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-02-21T23:38:24.746 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-02-21T23:38:24.747 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 62/162 2024-02-21T23:38:24.803 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : iproute-6.2.0-5.el8_9.x86_64 137/162 2024-02-21T23:38:24.980 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-02-21T23:38:24.980 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 152/162 2024-02-21T23:38:24.997 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 63/162 2024-02-21T23:38:25.033 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : xfsdump-3.1.8-7.el8_9.x86_64 138/162 2024-02-21T23:38:25.188 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 64/162 2024-02-21T23:38:25.206 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 152/162 2024-02-21T23:38:25.255 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : cephfs-top-2:18.2.1-638.gc1dc18f4.el8.noarch 139/162 2024-02-21T23:38:25.403 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 153/162 2024-02-21T23:38:25.454 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 65/162 2024-02-21T23:38:25.471 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-fuse-2:18.2.1-638.gc1dc18f4.el8.x86_64 140/162 2024-02-21T23:38:25.592 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : util-linux-user-2.32.1-35.el8.x86_64 154/162 2024-02-21T23:38:25.775 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-nbd-2:18.2.1-638.gc1dc18f4.el8.x86_64 141/162 2024-02-21T23:38:25.801 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 66/162 2024-02-21T23:38:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : util-linux-2.32.1-35.el8.x86_64 155/162 2024-02-21T23:38:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : libfdisk-2.32.1-35.el8.x86_64 156/162 2024-02-21T23:38:26.056 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : openssl-1:1.1.1k-12.el8_9.x86_64 142/162 2024-02-21T23:38:26.079 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libfdisk-2.32.1-35.el8.x86_64 156/162 2024-02-21T23:38:26.079 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : libmount-2.32.1-35.el8.x86_64 157/162 2024-02-21T23:38:26.101 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 67/162 2024-02-21T23:38:26.254 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : NetworkManager-initscripts-updown-1:1.40.16-13.e 143/162 2024-02-21T23:38:26.265 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-13.e 143/162 2024-02-21T23:38:26.304 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libmount-2.32.1-35.el8.x86_64 157/162 2024-02-21T23:38:26.304 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : libblkid-2.32.1-35.el8.x86_64 158/162 2024-02-21T23:38:26.309 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 68/162 2024-02-21T23:38:26.434 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 144/162 2024-02-21T23:38:26.529 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libblkid-2.32.1-35.el8.x86_64 158/162 2024-02-21T23:38:26.529 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : libuuid-2.32.1-35.el8.x86_64 159/162 2024-02-21T23:38:26.588 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 69/162 2024-02-21T23:38:26.662 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : flex-2.6.1-9.el8.x86_64 145/162 2024-02-21T23:38:26.744 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 145/162 2024-02-21T23:38:26.787 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libuuid-2.32.1-35.el8.x86_64 159/162 2024-02-21T23:38:26.787 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-02-21T23:38:26.847 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 70/162 2024-02-21T23:38:26.920 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : bison-3.0.4-10.el8.x86_64 146/162 2024-02-21T23:38:26.934 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 146/162 2024-02-21T23:38:27.029 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-02-21T23:38:27.128 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : elfutils-libs-0.186-1.el8.x86_64 147/162 2024-02-21T23:38:27.171 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : elfutils-libelf-0.186-1.el8.x86_64 161/162 2024-02-21T23:38:27.171 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-02-21T23:38:27.175 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 71/162 2024-02-21T23:38:27.321 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 148/162 2024-02-21T23:38:27.383 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 72/162 2024-02-21T23:38:27.491 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 149/162 2024-02-21T23:38:27.491 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-02-21T23:38:27.626 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 73/162 2024-02-21T23:38:27.790 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-02-21T23:38:27.790 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-02-21T23:38:27.861 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 74/162 2024-02-21T23:38:28.024 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-02-21T23:38:28.024 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 152/162 2024-02-21T23:38:28.089 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 75/162 2024-02-21T23:38:28.114 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-02-21T23:38:28.114 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-2:18.2.1-638.gc1dc18f4.el8.x86_64 1/162 2024-02-21T23:38:28.114 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-base-2:18.2.1-638.gc1dc18f4.el8.x86_64 2/162 2024-02-21T23:38:28.114 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-common-2:18.2.1-638.gc1dc18f4.el8.x86_64 3/162 2024-02-21T23:38:28.115 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-fuse-2:18.2.1-638.gc1dc18f4.el8.x86_64 4/162 2024-02-21T23:38:28.115 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-immutable-object-cache-2:18.2.1-638.gc1dc18 5/162 2024-02-21T23:38:28.115 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mds-2:18.2.1-638.gc1dc18f4.el8.x86_64 6/162 2024-02-21T23:38:28.115 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-2:18.2.1-638.gc1dc18f4.el8.x86_64 7/162 2024-02-21T23:38:28.115 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mon-2:18.2.1-638.gc1dc18f4.el8.x86_64 8/162 2024-02-21T23:38:28.115 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-osd-2:18.2.1-638.gc1dc18f4.el8.x86_64 9/162 2024-02-21T23:38:28.115 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-radosgw-2:18.2.1-638.gc1dc18f4.el8.x86_64 10/162 2024-02-21T23:38:28.115 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-selinux-2:18.2.1-638.gc1dc18f4.el8.x86_64 11/162 2024-02-21T23:38:28.115 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-test-2:18.2.1-638.gc1dc18f4.el8.x86_64 12/162 2024-02-21T23:38:28.115 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : cephfs-mirror-2:18.2.1-638.gc1dc18f4.el8.x86_64 13/162 2024-02-21T23:38:28.115 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcephfs-devel-2:18.2.1-638.gc1dc18f4.el8.x86_6 14/162 2024-02-21T23:38:28.115 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcephfs2-2:18.2.1-638.gc1dc18f4.el8.x86_64 15/162 2024-02-21T23:38:28.115 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcephsqlite-2:18.2.1-638.gc1dc18f4.el8.x86_64 16/162 2024-02-21T23:38:28.116 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librados-devel-2:18.2.1-638.gc1dc18f4.el8.x86_64 17/162 2024-02-21T23:38:28.116 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libradosstriper1-2:18.2.1-638.gc1dc18f4.el8.x86_ 18/162 2024-02-21T23:38:28.116 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librgw2-2:18.2.1-638.gc1dc18f4.el8.x86_64 19/162 2024-02-21T23:38:28.116 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-ceph-argparse-2:18.2.1-638.gc1dc18f4.el8 20/162 2024-02-21T23:38:28.116 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-ceph-common-2:18.2.1-638.gc1dc18f4.el8.x 21/162 2024-02-21T23:38:28.116 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cephfs-2:18.2.1-638.gc1dc18f4.el8.x86_64 22/162 2024-02-21T23:38:28.116 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rados-2:18.2.1-638.gc1dc18f4.el8.x86_64 23/162 2024-02-21T23:38:28.116 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rbd-2:18.2.1-638.gc1dc18f4.el8.x86_64 24/162 2024-02-21T23:38:28.116 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rgw-2:18.2.1-638.gc1dc18f4.el8.x86_64 25/162 2024-02-21T23:38:28.116 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-fuse-2:18.2.1-638.gc1dc18f4.el8.x86_64 26/162 2024-02-21T23:38:28.117 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-mirror-2:18.2.1-638.gc1dc18f4.el8.x86_64 27/162 2024-02-21T23:38:28.117 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-nbd-2:18.2.1-638.gc1dc18f4.el8.x86_64 28/162 2024-02-21T23:38:28.117 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-grafana-dashboards-2:18.2.1-638.gc1dc18f4.e 29/162 2024-02-21T23:38:28.117 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-cephadm-2:18.2.1-638.gc1dc18f4.el8.noar 30/162 2024-02-21T23:38:28.117 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-dashboard-2:18.2.1-638.gc1dc18f4.el8.no 31/162 2024-02-21T23:38:28.117 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.1-638.gc1dc 32/162 2024-02-21T23:38:28.117 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-modules-core-2:18.2.1-638.gc1dc18f4.el8 33/162 2024-02-21T23:38:28.117 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-rook-2:18.2.1-638.gc1dc18f4.el8.noarch 34/162 2024-02-21T23:38:28.117 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-prometheus-alerts-2:18.2.1-638.gc1dc18f4.el 35/162 2024-02-21T23:38:28.117 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-volume-2:18.2.1-638.gc1dc18f4.el8.noarch 36/162 2024-02-21T23:38:28.117 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : cephadm-2:18.2.1-638.gc1dc18f4.el8.noarch 37/162 2024-02-21T23:38:28.117 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : cephfs-top-2:18.2.1-638.gc1dc18f4.el8.noarch 38/162 2024-02-21T23:38:28.117 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/162 2024-02-21T23:38:28.118 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/162 2024-02-21T23:38:28.118 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/162 2024-02-21T23:38:28.118 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/162 2024-02-21T23:38:28.118 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/162 2024-02-21T23:38:28.118 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/162 2024-02-21T23:38:28.118 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/162 2024-02-21T23:38:28.118 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/162 2024-02-21T23:38:28.118 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/162 2024-02-21T23:38:28.118 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/162 2024-02-21T23:38:28.118 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/162 2024-02-21T23:38:28.118 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/162 2024-02-21T23:38:28.118 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/162 2024-02-21T23:38:28.118 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/162 2024-02-21T23:38:28.118 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/162 2024-02-21T23:38:28.119 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/162 2024-02-21T23:38:28.119 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/162 2024-02-21T23:38:28.119 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/162 2024-02-21T23:38:28.119 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/162 2024-02-21T23:38:28.119 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/162 2024-02-21T23:38:28.119 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/162 2024-02-21T23:38:28.119 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/162 2024-02-21T23:38:28.119 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/162 2024-02-21T23:38:28.119 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/162 2024-02-21T23:38:28.119 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/162 2024-02-21T23:38:28.119 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/162 2024-02-21T23:38:28.119 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/162 2024-02-21T23:38:28.120 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/162 2024-02-21T23:38:28.120 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/162 2024-02-21T23:38:28.120 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/162 2024-02-21T23:38:28.120 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/162 2024-02-21T23:38:28.120 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/162 2024-02-21T23:38:28.120 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/162 2024-02-21T23:38:28.120 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 72/162 2024-02-21T23:38:28.120 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 73/162 2024-02-21T23:38:28.120 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 74/162 2024-02-21T23:38:28.120 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 75/162 2024-02-21T23:38:28.120 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 76/162 2024-02-21T23:38:28.120 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 77/162 2024-02-21T23:38:28.120 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/162 2024-02-21T23:38:28.121 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : flex-2.6.1-9.el8.x86_64 79/162 2024-02-21T23:38:28.121 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 80/162 2024-02-21T23:38:28.121 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 81/162 2024-02-21T23:38:28.121 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 82/162 2024-02-21T23:38:28.121 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 83/162 2024-02-21T23:38:28.121 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 84/162 2024-02-21T23:38:28.121 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 85/162 2024-02-21T23:38:28.121 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 86/162 2024-02-21T23:38:28.122 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 87/162 2024-02-21T23:38:28.122 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 88/162 2024-02-21T23:38:28.122 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 89/162 2024-02-21T23:38:28.122 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 90/162 2024-02-21T23:38:28.122 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 91/162 2024-02-21T23:38:28.122 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 92/162 2024-02-21T23:38:28.122 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 93/162 2024-02-21T23:38:28.122 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 94/162 2024-02-21T23:38:28.122 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : automake-1.16.1-8.el8.noarch 95/162 2024-02-21T23:38:28.123 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libpq-13.11-1.el8.x86_64 96/162 2024-02-21T23:38:28.123 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : jq-1.6-7.el8.x86_64 97/162 2024-02-21T23:38:28.123 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 98/162 2024-02-21T23:38:28.123 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : postgresql-10.23-3.module+el8.9.0+20831+78489779 99/162 2024-02-21T23:38:28.123 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : postgresql-contrib-10.23-3.module+el8.9.0+20831+ 100/162 2024-02-21T23:38:28.123 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : postgresql-server-10.23-3.module+el8.9.0+20831+7 101/162 2024-02-21T23:38:28.123 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 102/162 2024-02-21T23:38:28.123 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 103/162 2024-02-21T23:38:28.123 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 104/162 2024-02-21T23:38:28.123 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/162 2024-02-21T23:38:28.123 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 106/162 2024-02-21T23:38:28.123 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 107/162 2024-02-21T23:38:28.123 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 108/162 2024-02-21T23:38:28.123 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 109/162 2024-02-21T23:38:28.124 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 110/162 2024-02-21T23:38:28.124 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 111/162 2024-02-21T23:38:28.124 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 112/162 2024-02-21T23:38:28.124 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 113/162 2024-02-21T23:38:28.124 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 114/162 2024-02-21T23:38:28.124 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 115/162 2024-02-21T23:38:28.124 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 116/162 2024-02-21T23:38:28.124 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 117/162 2024-02-21T23:38:28.124 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 118/162 2024-02-21T23:38:28.124 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 119/162 2024-02-21T23:38:28.124 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 120/162 2024-02-21T23:38:28.124 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 121/162 2024-02-21T23:38:28.124 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 122/162 2024-02-21T23:38:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 123/162 2024-02-21T23:38:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 124/162 2024-02-21T23:38:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 125/162 2024-02-21T23:38:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-13.e 126/162 2024-02-21T23:38:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 127/162 2024-02-21T23:38:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : krb5-devel-1.18.2-26.el8_9.x86_64 128/162 2024-02-21T23:38:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 129/162 2024-02-21T23:38:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8_9.x86_64 130/162 2024-02-21T23:38:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librados2-2:18.2.1-638.gc1dc18f4.el8.x86_64 131/162 2024-02-21T23:38:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 132/162 2024-02-21T23:38:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librbd1-2:18.2.1-638.gc1dc18f4.el8.x86_64 133/162 2024-02-21T23:38:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 134/162 2024-02-21T23:38:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 135/162 2024-02-21T23:38:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : elfutils-libelf-0.186-1.el8.x86_64 136/162 2024-02-21T23:38:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 137/162 2024-02-21T23:38:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : elfutils-libs-0.186-1.el8.x86_64 138/162 2024-02-21T23:38:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 139/162 2024-02-21T23:38:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libmount-2.32.1-35.el8.x86_64 140/162 2024-02-21T23:38:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 141/162 2024-02-21T23:38:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libsmartcols-2.32.1-35.el8.x86_64 142/162 2024-02-21T23:38:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 143/162 2024-02-21T23:38:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libuuid-2.32.1-35.el8.x86_64 144/162 2024-02-21T23:38:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 145/162 2024-02-21T23:38:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : util-linux-2.32.1-35.el8.x86_64 146/162 2024-02-21T23:38:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 147/162 2024-02-21T23:38:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : util-linux-user-2.32.1-35.el8.x86_64 148/162 2024-02-21T23:38:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : iproute-6.2.0-5.el8_9.x86_64 149/162 2024-02-21T23:38:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 150/162 2024-02-21T23:38:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 151/162 2024-02-21T23:38:28.127 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libblkid-2.32.1-35.el8.x86_64 152/162 2024-02-21T23:38:28.127 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 153/162 2024-02-21T23:38:28.127 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libfdisk-2.32.1-35.el8.x86_64 154/162 2024-02-21T23:38:28.127 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : openssl-1:1.1.1k-12.el8_9.x86_64 155/162 2024-02-21T23:38:28.127 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 156/162 2024-02-21T23:38:28.127 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8_9.x86_64 157/162 2024-02-21T23:38:28.127 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 158/162 2024-02-21T23:38:28.127 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : xfsdump-3.1.8-7.el8_9.x86_64 159/162 2024-02-21T23:38:28.127 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 160/162 2024-02-21T23:38:28.127 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 161/162 2024-02-21T23:38:28.276 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 152/162 2024-02-21T23:38:28.385 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : krb5-devel-1.18.2-26.el8_9.x86_64 76/162 2024-02-21T23:38:28.538 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 153/162 2024-02-21T23:38:28.576 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-packaging-16.8-9.el8.noarch 77/162 2024-02-21T23:38:28.696 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : util-linux-user-2.32.1-35.el8.x86_64 154/162 2024-02-21T23:38:28.784 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 78/162 2024-02-21T23:38:28.930 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : util-linux-2.32.1-35.el8.x86_64 155/162 2024-02-21T23:38:28.930 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : libfdisk-2.32.1-35.el8.x86_64 156/162 2024-02-21T23:38:28.981 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 79/162 2024-02-21T23:38:29.164 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libfdisk-2.32.1-35.el8.x86_64 156/162 2024-02-21T23:38:29.164 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : libmount-2.32.1-35.el8.x86_64 157/162 2024-02-21T23:38:29.314 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 80/162 2024-02-21T23:38:29.406 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libmount-2.32.1-35.el8.x86_64 157/162 2024-02-21T23:38:29.406 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : libblkid-2.32.1-35.el8.x86_64 158/162 2024-02-21T23:38:29.558 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 81/162 2024-02-21T23:38:29.606 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libblkid-2.32.1-35.el8.x86_64 158/162 2024-02-21T23:38:29.606 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : libuuid-2.32.1-35.el8.x86_64 159/162 2024-02-21T23:38:29.772 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 82/162 2024-02-21T23:38:29.847 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libuuid-2.32.1-35.el8.x86_64 159/162 2024-02-21T23:38:29.847 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-02-21T23:38:29.954 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 83/162 2024-02-21T23:38:30.072 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-02-21T23:38:30.203 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 84/162 2024-02-21T23:38:30.214 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : elfutils-libelf-0.186-1.el8.x86_64 161/162 2024-02-21T23:38:30.214 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-02-21T23:38:30.412 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 85/162 2024-02-21T23:38:30.795 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 86/162 2024-02-21T23:38:31.028 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 87/162 2024-02-21T23:38:31.123 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-02-21T23:38:31.124 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-2:18.2.1-638.gc1dc18f4.el8.x86_64 1/162 2024-02-21T23:38:31.124 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-base-2:18.2.1-638.gc1dc18f4.el8.x86_64 2/162 2024-02-21T23:38:31.124 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-common-2:18.2.1-638.gc1dc18f4.el8.x86_64 3/162 2024-02-21T23:38:31.124 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-fuse-2:18.2.1-638.gc1dc18f4.el8.x86_64 4/162 2024-02-21T23:38:31.124 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-immutable-object-cache-2:18.2.1-638.gc1dc18 5/162 2024-02-21T23:38:31.124 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mds-2:18.2.1-638.gc1dc18f4.el8.x86_64 6/162 2024-02-21T23:38:31.124 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-2:18.2.1-638.gc1dc18f4.el8.x86_64 7/162 2024-02-21T23:38:31.124 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mon-2:18.2.1-638.gc1dc18f4.el8.x86_64 8/162 2024-02-21T23:38:31.124 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-osd-2:18.2.1-638.gc1dc18f4.el8.x86_64 9/162 2024-02-21T23:38:31.124 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-radosgw-2:18.2.1-638.gc1dc18f4.el8.x86_64 10/162 2024-02-21T23:38:31.124 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-selinux-2:18.2.1-638.gc1dc18f4.el8.x86_64 11/162 2024-02-21T23:38:31.125 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-test-2:18.2.1-638.gc1dc18f4.el8.x86_64 12/162 2024-02-21T23:38:31.125 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : cephfs-mirror-2:18.2.1-638.gc1dc18f4.el8.x86_64 13/162 2024-02-21T23:38:31.125 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephfs-devel-2:18.2.1-638.gc1dc18f4.el8.x86_6 14/162 2024-02-21T23:38:31.125 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephfs2-2:18.2.1-638.gc1dc18f4.el8.x86_64 15/162 2024-02-21T23:38:31.125 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephsqlite-2:18.2.1-638.gc1dc18f4.el8.x86_64 16/162 2024-02-21T23:38:31.125 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados-devel-2:18.2.1-638.gc1dc18f4.el8.x86_64 17/162 2024-02-21T23:38:31.125 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libradosstriper1-2:18.2.1-638.gc1dc18f4.el8.x86_ 18/162 2024-02-21T23:38:31.125 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librgw2-2:18.2.1-638.gc1dc18f4.el8.x86_64 19/162 2024-02-21T23:38:31.125 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-ceph-argparse-2:18.2.1-638.gc1dc18f4.el8 20/162 2024-02-21T23:38:31.125 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-ceph-common-2:18.2.1-638.gc1dc18f4.el8.x 21/162 2024-02-21T23:38:31.125 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cephfs-2:18.2.1-638.gc1dc18f4.el8.x86_64 22/162 2024-02-21T23:38:31.125 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rados-2:18.2.1-638.gc1dc18f4.el8.x86_64 23/162 2024-02-21T23:38:31.125 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rbd-2:18.2.1-638.gc1dc18f4.el8.x86_64 24/162 2024-02-21T23:38:31.125 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rgw-2:18.2.1-638.gc1dc18f4.el8.x86_64 25/162 2024-02-21T23:38:31.125 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-fuse-2:18.2.1-638.gc1dc18f4.el8.x86_64 26/162 2024-02-21T23:38:31.126 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-mirror-2:18.2.1-638.gc1dc18f4.el8.x86_64 27/162 2024-02-21T23:38:31.126 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-nbd-2:18.2.1-638.gc1dc18f4.el8.x86_64 28/162 2024-02-21T23:38:31.126 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-grafana-dashboards-2:18.2.1-638.gc1dc18f4.e 29/162 2024-02-21T23:38:31.126 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-cephadm-2:18.2.1-638.gc1dc18f4.el8.noar 30/162 2024-02-21T23:38:31.126 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-dashboard-2:18.2.1-638.gc1dc18f4.el8.no 31/162 2024-02-21T23:38:31.126 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.1-638.gc1dc 32/162 2024-02-21T23:38:31.126 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-modules-core-2:18.2.1-638.gc1dc18f4.el8 33/162 2024-02-21T23:38:31.126 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-rook-2:18.2.1-638.gc1dc18f4.el8.noarch 34/162 2024-02-21T23:38:31.126 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-prometheus-alerts-2:18.2.1-638.gc1dc18f4.el 35/162 2024-02-21T23:38:31.126 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-volume-2:18.2.1-638.gc1dc18f4.el8.noarch 36/162 2024-02-21T23:38:31.126 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : cephadm-2:18.2.1-638.gc1dc18f4.el8.noarch 37/162 2024-02-21T23:38:31.126 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : cephfs-top-2:18.2.1-638.gc1dc18f4.el8.noarch 38/162 2024-02-21T23:38:31.126 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/162 2024-02-21T23:38:31.126 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/162 2024-02-21T23:38:31.126 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/162 2024-02-21T23:38:31.127 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/162 2024-02-21T23:38:31.127 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/162 2024-02-21T23:38:31.127 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/162 2024-02-21T23:38:31.127 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/162 2024-02-21T23:38:31.127 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/162 2024-02-21T23:38:31.127 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/162 2024-02-21T23:38:31.127 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/162 2024-02-21T23:38:31.127 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/162 2024-02-21T23:38:31.127 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/162 2024-02-21T23:38:31.127 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/162 2024-02-21T23:38:31.127 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/162 2024-02-21T23:38:31.127 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/162 2024-02-21T23:38:31.127 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/162 2024-02-21T23:38:31.127 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/162 2024-02-21T23:38:31.127 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/162 2024-02-21T23:38:31.127 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/162 2024-02-21T23:38:31.128 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/162 2024-02-21T23:38:31.128 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/162 2024-02-21T23:38:31.128 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/162 2024-02-21T23:38:31.128 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/162 2024-02-21T23:38:31.128 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/162 2024-02-21T23:38:31.128 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/162 2024-02-21T23:38:31.128 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/162 2024-02-21T23:38:31.128 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/162 2024-02-21T23:38:31.128 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/162 2024-02-21T23:38:31.128 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/162 2024-02-21T23:38:31.128 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/162 2024-02-21T23:38:31.128 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/162 2024-02-21T23:38:31.128 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/162 2024-02-21T23:38:31.128 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/162 2024-02-21T23:38:31.129 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 72/162 2024-02-21T23:38:31.129 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 73/162 2024-02-21T23:38:31.129 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 74/162 2024-02-21T23:38:31.129 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 75/162 2024-02-21T23:38:31.129 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 76/162 2024-02-21T23:38:31.129 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 77/162 2024-02-21T23:38:31.129 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/162 2024-02-21T23:38:31.129 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : flex-2.6.1-9.el8.x86_64 79/162 2024-02-21T23:38:31.129 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 80/162 2024-02-21T23:38:31.129 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 81/162 2024-02-21T23:38:31.129 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 82/162 2024-02-21T23:38:31.129 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 83/162 2024-02-21T23:38:31.129 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 84/162 2024-02-21T23:38:31.129 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 85/162 2024-02-21T23:38:31.130 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 86/162 2024-02-21T23:38:31.130 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 87/162 2024-02-21T23:38:31.130 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 88/162 2024-02-21T23:38:31.130 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 89/162 2024-02-21T23:38:31.130 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 90/162 2024-02-21T23:38:31.130 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 91/162 2024-02-21T23:38:31.130 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 92/162 2024-02-21T23:38:31.130 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 93/162 2024-02-21T23:38:31.130 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 94/162 2024-02-21T23:38:31.130 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : automake-1.16.1-8.el8.noarch 95/162 2024-02-21T23:38:31.130 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libpq-13.11-1.el8.x86_64 96/162 2024-02-21T23:38:31.130 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : jq-1.6-7.el8.x86_64 97/162 2024-02-21T23:38:31.131 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 98/162 2024-02-21T23:38:31.131 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : postgresql-10.23-3.module+el8.9.0+20831+78489779 99/162 2024-02-21T23:38:31.131 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : postgresql-contrib-10.23-3.module+el8.9.0+20831+ 100/162 2024-02-21T23:38:31.131 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : postgresql-server-10.23-3.module+el8.9.0+20831+7 101/162 2024-02-21T23:38:31.131 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 102/162 2024-02-21T23:38:31.131 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 103/162 2024-02-21T23:38:31.131 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 104/162 2024-02-21T23:38:31.131 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/162 2024-02-21T23:38:31.131 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 106/162 2024-02-21T23:38:31.131 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 107/162 2024-02-21T23:38:31.131 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 108/162 2024-02-21T23:38:31.131 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 109/162 2024-02-21T23:38:31.131 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 110/162 2024-02-21T23:38:31.131 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 111/162 2024-02-21T23:38:31.132 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 112/162 2024-02-21T23:38:31.132 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 113/162 2024-02-21T23:38:31.132 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 114/162 2024-02-21T23:38:31.132 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 115/162 2024-02-21T23:38:31.132 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 116/162 2024-02-21T23:38:31.132 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 117/162 2024-02-21T23:38:31.132 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 118/162 2024-02-21T23:38:31.132 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 119/162 2024-02-21T23:38:31.132 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 120/162 2024-02-21T23:38:31.132 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 121/162 2024-02-21T23:38:31.132 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 122/162 2024-02-21T23:38:31.132 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 123/162 2024-02-21T23:38:31.132 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 124/162 2024-02-21T23:38:31.133 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 125/162 2024-02-21T23:38:31.133 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-13.e 126/162 2024-02-21T23:38:31.133 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 127/162 2024-02-21T23:38:31.133 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : krb5-devel-1.18.2-26.el8_9.x86_64 128/162 2024-02-21T23:38:31.133 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 129/162 2024-02-21T23:38:31.133 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8_9.x86_64 130/162 2024-02-21T23:38:31.133 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados2-2:18.2.1-638.gc1dc18f4.el8.x86_64 131/162 2024-02-21T23:38:31.133 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 132/162 2024-02-21T23:38:31.133 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librbd1-2:18.2.1-638.gc1dc18f4.el8.x86_64 133/162 2024-02-21T23:38:31.133 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 134/162 2024-02-21T23:38:31.133 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 135/162 2024-02-21T23:38:31.133 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : elfutils-libelf-0.186-1.el8.x86_64 136/162 2024-02-21T23:38:31.133 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 137/162 2024-02-21T23:38:31.133 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : elfutils-libs-0.186-1.el8.x86_64 138/162 2024-02-21T23:38:31.134 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 139/162 2024-02-21T23:38:31.134 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libmount-2.32.1-35.el8.x86_64 140/162 2024-02-21T23:38:31.134 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 141/162 2024-02-21T23:38:31.134 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libsmartcols-2.32.1-35.el8.x86_64 142/162 2024-02-21T23:38:31.134 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 143/162 2024-02-21T23:38:31.134 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libuuid-2.32.1-35.el8.x86_64 144/162 2024-02-21T23:38:31.134 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 145/162 2024-02-21T23:38:31.134 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : util-linux-2.32.1-35.el8.x86_64 146/162 2024-02-21T23:38:31.134 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 147/162 2024-02-21T23:38:31.134 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : util-linux-user-2.32.1-35.el8.x86_64 148/162 2024-02-21T23:38:31.134 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : iproute-6.2.0-5.el8_9.x86_64 149/162 2024-02-21T23:38:31.134 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 150/162 2024-02-21T23:38:31.134 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 151/162 2024-02-21T23:38:31.134 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libblkid-2.32.1-35.el8.x86_64 152/162 2024-02-21T23:38:31.134 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 153/162 2024-02-21T23:38:31.134 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libfdisk-2.32.1-35.el8.x86_64 154/162 2024-02-21T23:38:31.135 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openssl-1:1.1.1k-12.el8_9.x86_64 155/162 2024-02-21T23:38:31.135 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 156/162 2024-02-21T23:38:31.135 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8_9.x86_64 157/162 2024-02-21T23:38:31.135 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 158/162 2024-02-21T23:38:31.135 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : xfsdump-3.1.8-7.el8_9.x86_64 159/162 2024-02-21T23:38:31.135 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 160/162 2024-02-21T23:38:31.135 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 161/162 2024-02-21T23:38:31.348 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 88/162 2024-02-21T23:38:31.604 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 89/162 2024-02-21T23:38:31.939 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 90/162 2024-02-21T23:38:32.148 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 91/162 2024-02-21T23:38:32.470 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-portend-2.6-1.el8.noarch 92/162 2024-02-21T23:38:32.691 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 93/162 2024-02-21T23:38:33.049 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 94/162 2024-02-21T23:38:33.332 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 95/162 2024-02-21T23:38:34.593 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 96/162 2024-02-21T23:38:34.945 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 97/162 2024-02-21T23:38:35.367 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 98/162 2024-02-21T23:38:35.652 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 99/162 2024-02-21T23:38:36.059 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 100/162 2024-02-21T23:38:36.249 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 101/162 2024-02-21T23:38:37.196 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 102/162 2024-02-21T23:38:37.425 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librgw2-2:18.2.1-638.gc1dc18f4.el8.x86_64 103/162 2024-02-21T23:38:37.500 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librgw2-2:18.2.1-638.gc1dc18f4.el8.x86_64 103/162 2024-02-21T23:38:37.850 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rgw-2:18.2.1-638.gc1dc18f4.el8.x86_64 104/162 2024-02-21T23:38:42.144 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 162/162 2024-02-21T23:38:42.144 INFO:teuthology.orchestra.run.smithi176.stdout:Installed products updated. 2024-02-21T23:38:42.505 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-common-2:18.2.1-638.gc1dc18f4.el8.x86_64 105/162 2024-02-21T23:38:42.738 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-21T23:38:42.738 INFO:teuthology.orchestra.run.smithi176.stdout:Upgraded: 2024-02-21T23:38:42.739 INFO:teuthology.orchestra.run.smithi176.stdout: elfutils-libelf-0.189-3.el8.x86_64 2024-02-21T23:38:42.739 INFO:teuthology.orchestra.run.smithi176.stdout: elfutils-libs-0.189-3.el8.x86_64 2024-02-21T23:38:42.739 INFO:teuthology.orchestra.run.smithi176.stdout: iproute-6.2.0-5.el8_9.x86_64 2024-02-21T23:38:42.739 INFO:teuthology.orchestra.run.smithi176.stdout: libblkid-2.32.1-43.el8.x86_64 2024-02-21T23:38:42.739 INFO:teuthology.orchestra.run.smithi176.stdout: libfdisk-2.32.1-43.el8.x86_64 2024-02-21T23:38:42.739 INFO:teuthology.orchestra.run.smithi176.stdout: libmount-2.32.1-43.el8.x86_64 2024-02-21T23:38:42.739 INFO:teuthology.orchestra.run.smithi176.stdout: librados2-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:42.739 INFO:teuthology.orchestra.run.smithi176.stdout: librbd1-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:42.739 INFO:teuthology.orchestra.run.smithi176.stdout: libsmartcols-2.32.1-43.el8.x86_64 2024-02-21T23:38:42.739 INFO:teuthology.orchestra.run.smithi176.stdout: libuuid-2.32.1-43.el8.x86_64 2024-02-21T23:38:42.739 INFO:teuthology.orchestra.run.smithi176.stdout: openssl-1:1.1.1k-12.el8_9.x86_64 2024-02-21T23:38:42.739 INFO:teuthology.orchestra.run.smithi176.stdout: openssl-libs-1:1.1.1k-12.el8_9.x86_64 2024-02-21T23:38:42.739 INFO:teuthology.orchestra.run.smithi176.stdout: util-linux-2.32.1-43.el8.x86_64 2024-02-21T23:38:42.740 INFO:teuthology.orchestra.run.smithi176.stdout: util-linux-user-2.32.1-43.el8.x86_64 2024-02-21T23:38:42.740 INFO:teuthology.orchestra.run.smithi176.stdout: xfsdump-3.1.8-7.el8_9.x86_64 2024-02-21T23:38:42.740 INFO:teuthology.orchestra.run.smithi176.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-02-21T23:38:42.740 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-02-21T23:38:42.740 INFO:teuthology.orchestra.run.smithi176.stdout: NetworkManager-initscripts-updown-1:1.40.16-13.el8_9.noarch 2024-02-21T23:38:42.740 INFO:teuthology.orchestra.run.smithi176.stdout: automake-1.16.1-8.el8.noarch 2024-02-21T23:38:42.740 INFO:teuthology.orchestra.run.smithi176.stdout: bison-3.0.4-10.el8.x86_64 2024-02-21T23:38:42.740 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:42.740 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-base-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:42.740 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-common-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:42.740 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-fuse-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:42.740 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-grafana-dashboards-2:18.2.1-638.gc1dc18f4.el8.noarch 2024-02-21T23:38:42.740 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-immutable-object-cache-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:42.741 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mds-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:42.741 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:42.741 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-cephadm-2:18.2.1-638.gc1dc18f4.el8.noarch 2024-02-21T23:38:42.741 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-dashboard-2:18.2.1-638.gc1dc18f4.el8.noarch 2024-02-21T23:38:42.741 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-diskprediction-local-2:18.2.1-638.gc1dc18f4.el8.noarch 2024-02-21T23:38:42.741 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-modules-core-2:18.2.1-638.gc1dc18f4.el8.noarch 2024-02-21T23:38:42.741 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-rook-2:18.2.1-638.gc1dc18f4.el8.noarch 2024-02-21T23:38:42.741 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mon-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:42.741 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-osd-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:42.741 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-prometheus-alerts-2:18.2.1-638.gc1dc18f4.el8.noarch 2024-02-21T23:38:42.741 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-radosgw-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:42.741 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-selinux-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:42.741 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-test-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:42.741 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-volume-2:18.2.1-638.gc1dc18f4.el8.noarch 2024-02-21T23:38:42.742 INFO:teuthology.orchestra.run.smithi176.stdout: cephadm-2:18.2.1-638.gc1dc18f4.el8.noarch 2024-02-21T23:38:42.742 INFO:teuthology.orchestra.run.smithi176.stdout: cephfs-mirror-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:42.742 INFO:teuthology.orchestra.run.smithi176.stdout: cephfs-top-2:18.2.1-638.gc1dc18f4.el8.noarch 2024-02-21T23:38:42.742 INFO:teuthology.orchestra.run.smithi176.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2024-02-21T23:38:42.742 INFO:teuthology.orchestra.run.smithi176.stdout: flex-2.6.1-9.el8.x86_64 2024-02-21T23:38:42.742 INFO:teuthology.orchestra.run.smithi176.stdout: fuse-2.9.7-17.el8.x86_64 2024-02-21T23:38:42.742 INFO:teuthology.orchestra.run.smithi176.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-21T23:38:42.742 INFO:teuthology.orchestra.run.smithi176.stdout: jq-1.6-7.el8.x86_64 2024-02-21T23:38:42.742 INFO:teuthology.orchestra.run.smithi176.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-02-21T23:38:42.742 INFO:teuthology.orchestra.run.smithi176.stdout: krb5-devel-1.18.2-26.el8_9.x86_64 2024-02-21T23:38:42.742 INFO:teuthology.orchestra.run.smithi176.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-02-21T23:38:42.742 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs-devel-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:42.742 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs2-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:42.743 INFO:teuthology.orchestra.run.smithi176.stdout: libcephsqlite-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:42.743 INFO:teuthology.orchestra.run.smithi176.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2024-02-21T23:38:42.743 INFO:teuthology.orchestra.run.smithi176.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-02-21T23:38:42.743 INFO:teuthology.orchestra.run.smithi176.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-21T23:38:42.743 INFO:teuthology.orchestra.run.smithi176.stdout: libpq-13.11-1.el8.x86_64 2024-02-21T23:38:42.743 INFO:teuthology.orchestra.run.smithi176.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-02-21T23:38:42.743 INFO:teuthology.orchestra.run.smithi176.stdout: librabbitmq-0.9.0-4.el8.x86_64 2024-02-21T23:38:42.743 INFO:teuthology.orchestra.run.smithi176.stdout: librados-devel-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:42.743 INFO:teuthology.orchestra.run.smithi176.stdout: libradosstriper1-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:42.743 INFO:teuthology.orchestra.run.smithi176.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-02-21T23:38:42.743 INFO:teuthology.orchestra.run.smithi176.stdout: librgw2-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:42.743 INFO:teuthology.orchestra.run.smithi176.stdout: libselinux-devel-2.9-5.el8.x86_64 2024-02-21T23:38:42.743 INFO:teuthology.orchestra.run.smithi176.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-02-21T23:38:42.743 INFO:teuthology.orchestra.run.smithi176.stdout: libtool-2.4.6-25.el8.x86_64 2024-02-21T23:38:42.743 INFO:teuthology.orchestra.run.smithi176.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-21T23:38:42.744 INFO:teuthology.orchestra.run.smithi176.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2024-02-21T23:38:42.744 INFO:teuthology.orchestra.run.smithi176.stdout: libverto-devel-0.3.0-5.el8.x86_64 2024-02-21T23:38:42.744 INFO:teuthology.orchestra.run.smithi176.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-02-21T23:38:42.744 INFO:teuthology.orchestra.run.smithi176.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-21T23:38:42.744 INFO:teuthology.orchestra.run.smithi176.stdout: oniguruma-6.8.2-2.el8.x86_64 2024-02-21T23:38:42.744 INFO:teuthology.orchestra.run.smithi176.stdout: openblas-0.3.15-6.el8.x86_64 2024-02-21T23:38:42.744 INFO:teuthology.orchestra.run.smithi176.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-02-21T23:38:42.744 INFO:teuthology.orchestra.run.smithi176.stdout: openssl-devel-1:1.1.1k-12.el8_9.x86_64 2024-02-21T23:38:42.744 INFO:teuthology.orchestra.run.smithi176.stdout: pcre2-devel-10.32-2.el8.x86_64 2024-02-21T23:38:42.744 INFO:teuthology.orchestra.run.smithi176.stdout: pcre2-utf16-10.32-2.el8.x86_64 2024-02-21T23:38:42.744 INFO:teuthology.orchestra.run.smithi176.stdout: pcre2-utf32-10.32-2.el8.x86_64 2024-02-21T23:38:42.744 INFO:teuthology.orchestra.run.smithi176.stdout: postgresql-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-02-21T23:38:42.745 INFO:teuthology.orchestra.run.smithi176.stdout: postgresql-contrib-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-02-21T23:38:42.745 INFO:teuthology.orchestra.run.smithi176.stdout: postgresql-server-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-02-21T23:38:42.745 INFO:teuthology.orchestra.run.smithi176.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-02-21T23:38:42.745 INFO:teuthology.orchestra.run.smithi176.stdout: python3-babel-2.5.1-7.el8.noarch 2024-02-21T23:38:42.745 INFO:teuthology.orchestra.run.smithi176.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-21T23:38:42.745 INFO:teuthology.orchestra.run.smithi176.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-21T23:38:42.745 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-21T23:38:42.745 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-argparse-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:42.746 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-common-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:42.746 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cephfs-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:42.746 INFO:teuthology.orchestra.run.smithi176.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-21T23:38:42.746 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-21T23:38:42.746 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-21T23:38:42.746 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-21T23:38:42.746 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2024-02-21T23:38:42.746 INFO:teuthology.orchestra.run.smithi176.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-21T23:38:42.746 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-21T23:38:42.746 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-21T23:38:42.746 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-02-21T23:38:42.746 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-21T23:38:42.746 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-21T23:38:42.746 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-21T23:38:42.746 INFO:teuthology.orchestra.run.smithi176.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-21T23:38:42.747 INFO:teuthology.orchestra.run.smithi176.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-21T23:38:42.747 INFO:teuthology.orchestra.run.smithi176.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-21T23:38:42.747 INFO:teuthology.orchestra.run.smithi176.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-21T23:38:42.747 INFO:teuthology.orchestra.run.smithi176.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-21T23:38:42.747 INFO:teuthology.orchestra.run.smithi176.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-02-21T23:38:42.747 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-21T23:38:42.747 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-21T23:38:42.747 INFO:teuthology.orchestra.run.smithi176.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-21T23:38:42.747 INFO:teuthology.orchestra.run.smithi176.stdout: python3-packaging-16.8-9.el8.noarch 2024-02-21T23:38:42.747 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-21T23:38:42.747 INFO:teuthology.orchestra.run.smithi176.stdout: python3-portend-2.6-1.el8.noarch 2024-02-21T23:38:42.747 INFO:teuthology.orchestra.run.smithi176.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-21T23:38:42.747 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-21T23:38:42.747 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-21T23:38:42.748 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-21T23:38:42.748 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-21T23:38:42.748 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-21T23:38:42.748 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rados-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:42.748 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rbd-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:42.748 INFO:teuthology.orchestra.run.smithi176.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-21T23:38:42.748 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-21T23:38:42.748 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rgw-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:42.748 INFO:teuthology.orchestra.run.smithi176.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-21T23:38:42.748 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-21T23:38:42.748 INFO:teuthology.orchestra.run.smithi176.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-02-21T23:38:42.748 INFO:teuthology.orchestra.run.smithi176.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-21T23:38:42.748 INFO:teuthology.orchestra.run.smithi176.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-21T23:38:42.749 INFO:teuthology.orchestra.run.smithi176.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-21T23:38:42.749 INFO:teuthology.orchestra.run.smithi176.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-21T23:38:42.749 INFO:teuthology.orchestra.run.smithi176.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-21T23:38:42.749 INFO:teuthology.orchestra.run.smithi176.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-21T23:38:42.749 INFO:teuthology.orchestra.run.smithi176.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-21T23:38:42.749 INFO:teuthology.orchestra.run.smithi176.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-21T23:38:42.749 INFO:teuthology.orchestra.run.smithi176.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-21T23:38:42.749 INFO:teuthology.orchestra.run.smithi176.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-21T23:38:42.749 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-fuse-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:42.749 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-mirror-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:42.749 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-nbd-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:42.749 INFO:teuthology.orchestra.run.smithi176.stdout: socat-1.7.4.1-1.el8.x86_64 2024-02-21T23:38:42.749 INFO:teuthology.orchestra.run.smithi176.stdout: thrift-0.13.0-2.el8.x86_64 2024-02-21T23:38:42.749 INFO:teuthology.orchestra.run.smithi176.stdout: uuid-1.6.2-43.el8.x86_64 2024-02-21T23:38:42.749 INFO:teuthology.orchestra.run.smithi176.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-02-21T23:38:42.750 INFO:teuthology.orchestra.run.smithi176.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-02-21T23:38:42.750 INFO:teuthology.orchestra.run.smithi176.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2024-02-21T23:38:42.750 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-21T23:38:42.750 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-02-21T23:38:42.775 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-common-2:18.2.1-638.gc1dc18f4.el8.x86_64 105/162 2024-02-21T23:38:42.950 DEBUG:teuthology.parallel:result is None 2024-02-21T23:38:43.356 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-common-2:18.2.1-638.gc1dc18f4.el8.x86_64 105/162 2024-02-21T23:38:43.696 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-base-2:18.2.1-638.gc1dc18f4.el8.x86_64 106/162 2024-02-21T23:38:43.783 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-base-2:18.2.1-638.gc1dc18f4.el8.x86_64 106/162 2024-02-21T23:38:43.924 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-selinux-2:18.2.1-638.gc1dc18f4.el8.x86_64 107/162 2024-02-21T23:38:45.974 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 162/162 2024-02-21T23:38:45.974 INFO:teuthology.orchestra.run.smithi178.stdout:Installed products updated. 2024-02-21T23:38:46.574 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-21T23:38:46.574 INFO:teuthology.orchestra.run.smithi178.stdout:Upgraded: 2024-02-21T23:38:46.574 INFO:teuthology.orchestra.run.smithi178.stdout: elfutils-libelf-0.189-3.el8.x86_64 2024-02-21T23:38:46.574 INFO:teuthology.orchestra.run.smithi178.stdout: elfutils-libs-0.189-3.el8.x86_64 2024-02-21T23:38:46.574 INFO:teuthology.orchestra.run.smithi178.stdout: iproute-6.2.0-5.el8_9.x86_64 2024-02-21T23:38:46.574 INFO:teuthology.orchestra.run.smithi178.stdout: libblkid-2.32.1-43.el8.x86_64 2024-02-21T23:38:46.574 INFO:teuthology.orchestra.run.smithi178.stdout: libfdisk-2.32.1-43.el8.x86_64 2024-02-21T23:38:46.574 INFO:teuthology.orchestra.run.smithi178.stdout: libmount-2.32.1-43.el8.x86_64 2024-02-21T23:38:46.574 INFO:teuthology.orchestra.run.smithi178.stdout: librados2-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:46.574 INFO:teuthology.orchestra.run.smithi178.stdout: librbd1-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:46.574 INFO:teuthology.orchestra.run.smithi178.stdout: libsmartcols-2.32.1-43.el8.x86_64 2024-02-21T23:38:46.575 INFO:teuthology.orchestra.run.smithi178.stdout: libuuid-2.32.1-43.el8.x86_64 2024-02-21T23:38:46.575 INFO:teuthology.orchestra.run.smithi178.stdout: openssl-1:1.1.1k-12.el8_9.x86_64 2024-02-21T23:38:46.575 INFO:teuthology.orchestra.run.smithi178.stdout: openssl-libs-1:1.1.1k-12.el8_9.x86_64 2024-02-21T23:38:46.575 INFO:teuthology.orchestra.run.smithi178.stdout: util-linux-2.32.1-43.el8.x86_64 2024-02-21T23:38:46.575 INFO:teuthology.orchestra.run.smithi178.stdout: util-linux-user-2.32.1-43.el8.x86_64 2024-02-21T23:38:46.575 INFO:teuthology.orchestra.run.smithi178.stdout: xfsdump-3.1.8-7.el8_9.x86_64 2024-02-21T23:38:46.575 INFO:teuthology.orchestra.run.smithi178.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-02-21T23:38:46.575 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-02-21T23:38:46.575 INFO:teuthology.orchestra.run.smithi178.stdout: NetworkManager-initscripts-updown-1:1.40.16-13.el8_9.noarch 2024-02-21T23:38:46.575 INFO:teuthology.orchestra.run.smithi178.stdout: automake-1.16.1-8.el8.noarch 2024-02-21T23:38:46.575 INFO:teuthology.orchestra.run.smithi178.stdout: bison-3.0.4-10.el8.x86_64 2024-02-21T23:38:46.575 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:46.575 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-base-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:46.576 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-common-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:46.576 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-fuse-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:46.576 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-grafana-dashboards-2:18.2.1-638.gc1dc18f4.el8.noarch 2024-02-21T23:38:46.576 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-immutable-object-cache-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:46.576 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mds-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:46.576 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:46.576 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-cephadm-2:18.2.1-638.gc1dc18f4.el8.noarch 2024-02-21T23:38:46.576 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-dashboard-2:18.2.1-638.gc1dc18f4.el8.noarch 2024-02-21T23:38:46.576 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-diskprediction-local-2:18.2.1-638.gc1dc18f4.el8.noarch 2024-02-21T23:38:46.576 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-modules-core-2:18.2.1-638.gc1dc18f4.el8.noarch 2024-02-21T23:38:46.576 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-rook-2:18.2.1-638.gc1dc18f4.el8.noarch 2024-02-21T23:38:46.576 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mon-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:46.577 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-osd-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:46.577 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-prometheus-alerts-2:18.2.1-638.gc1dc18f4.el8.noarch 2024-02-21T23:38:46.577 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-radosgw-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:46.577 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-selinux-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:46.577 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-test-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:46.577 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-volume-2:18.2.1-638.gc1dc18f4.el8.noarch 2024-02-21T23:38:46.577 INFO:teuthology.orchestra.run.smithi178.stdout: cephadm-2:18.2.1-638.gc1dc18f4.el8.noarch 2024-02-21T23:38:46.577 INFO:teuthology.orchestra.run.smithi178.stdout: cephfs-mirror-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:46.577 INFO:teuthology.orchestra.run.smithi178.stdout: cephfs-top-2:18.2.1-638.gc1dc18f4.el8.noarch 2024-02-21T23:38:46.577 INFO:teuthology.orchestra.run.smithi178.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2024-02-21T23:38:46.577 INFO:teuthology.orchestra.run.smithi178.stdout: flex-2.6.1-9.el8.x86_64 2024-02-21T23:38:46.577 INFO:teuthology.orchestra.run.smithi178.stdout: fuse-2.9.7-17.el8.x86_64 2024-02-21T23:38:46.577 INFO:teuthology.orchestra.run.smithi178.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-21T23:38:46.578 INFO:teuthology.orchestra.run.smithi178.stdout: jq-1.6-7.el8.x86_64 2024-02-21T23:38:46.578 INFO:teuthology.orchestra.run.smithi178.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-02-21T23:38:46.578 INFO:teuthology.orchestra.run.smithi178.stdout: krb5-devel-1.18.2-26.el8_9.x86_64 2024-02-21T23:38:46.578 INFO:teuthology.orchestra.run.smithi178.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-02-21T23:38:46.578 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs-devel-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:46.578 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs2-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:46.578 INFO:teuthology.orchestra.run.smithi178.stdout: libcephsqlite-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:46.578 INFO:teuthology.orchestra.run.smithi178.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2024-02-21T23:38:46.578 INFO:teuthology.orchestra.run.smithi178.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-02-21T23:38:46.578 INFO:teuthology.orchestra.run.smithi178.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-21T23:38:46.578 INFO:teuthology.orchestra.run.smithi178.stdout: libpq-13.11-1.el8.x86_64 2024-02-21T23:38:46.578 INFO:teuthology.orchestra.run.smithi178.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-02-21T23:38:46.578 INFO:teuthology.orchestra.run.smithi178.stdout: librabbitmq-0.9.0-4.el8.x86_64 2024-02-21T23:38:46.578 INFO:teuthology.orchestra.run.smithi178.stdout: librados-devel-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:46.579 INFO:teuthology.orchestra.run.smithi178.stdout: libradosstriper1-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:46.579 INFO:teuthology.orchestra.run.smithi178.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-02-21T23:38:46.579 INFO:teuthology.orchestra.run.smithi178.stdout: librgw2-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:46.579 INFO:teuthology.orchestra.run.smithi178.stdout: libselinux-devel-2.9-5.el8.x86_64 2024-02-21T23:38:46.579 INFO:teuthology.orchestra.run.smithi178.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-02-21T23:38:46.579 INFO:teuthology.orchestra.run.smithi178.stdout: libtool-2.4.6-25.el8.x86_64 2024-02-21T23:38:46.579 INFO:teuthology.orchestra.run.smithi178.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-21T23:38:46.579 INFO:teuthology.orchestra.run.smithi178.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2024-02-21T23:38:46.579 INFO:teuthology.orchestra.run.smithi178.stdout: libverto-devel-0.3.0-5.el8.x86_64 2024-02-21T23:38:46.579 INFO:teuthology.orchestra.run.smithi178.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-02-21T23:38:46.579 INFO:teuthology.orchestra.run.smithi178.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-21T23:38:46.579 INFO:teuthology.orchestra.run.smithi178.stdout: oniguruma-6.8.2-2.el8.x86_64 2024-02-21T23:38:46.579 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-0.3.15-6.el8.x86_64 2024-02-21T23:38:46.579 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-02-21T23:38:46.580 INFO:teuthology.orchestra.run.smithi178.stdout: openssl-devel-1:1.1.1k-12.el8_9.x86_64 2024-02-21T23:38:46.580 INFO:teuthology.orchestra.run.smithi178.stdout: pcre2-devel-10.32-2.el8.x86_64 2024-02-21T23:38:46.580 INFO:teuthology.orchestra.run.smithi178.stdout: pcre2-utf16-10.32-2.el8.x86_64 2024-02-21T23:38:46.580 INFO:teuthology.orchestra.run.smithi178.stdout: pcre2-utf32-10.32-2.el8.x86_64 2024-02-21T23:38:46.580 INFO:teuthology.orchestra.run.smithi178.stdout: postgresql-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-02-21T23:38:46.580 INFO:teuthology.orchestra.run.smithi178.stdout: postgresql-contrib-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-02-21T23:38:46.580 INFO:teuthology.orchestra.run.smithi178.stdout: postgresql-server-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-02-21T23:38:46.580 INFO:teuthology.orchestra.run.smithi178.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-02-21T23:38:46.580 INFO:teuthology.orchestra.run.smithi178.stdout: python3-babel-2.5.1-7.el8.noarch 2024-02-21T23:38:46.580 INFO:teuthology.orchestra.run.smithi178.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-21T23:38:46.580 INFO:teuthology.orchestra.run.smithi178.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-21T23:38:46.580 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-21T23:38:46.580 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-argparse-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:46.581 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-common-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:46.581 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cephfs-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:46.581 INFO:teuthology.orchestra.run.smithi178.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-21T23:38:46.581 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-21T23:38:46.581 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-21T23:38:46.581 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-21T23:38:46.581 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2024-02-21T23:38:46.581 INFO:teuthology.orchestra.run.smithi178.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-21T23:38:46.581 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-21T23:38:46.581 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-21T23:38:46.581 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-02-21T23:38:46.581 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-21T23:38:46.581 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-21T23:38:46.581 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-21T23:38:46.582 INFO:teuthology.orchestra.run.smithi178.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-21T23:38:46.582 INFO:teuthology.orchestra.run.smithi178.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-21T23:38:46.582 INFO:teuthology.orchestra.run.smithi178.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-21T23:38:46.582 INFO:teuthology.orchestra.run.smithi178.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-21T23:38:46.582 INFO:teuthology.orchestra.run.smithi178.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-21T23:38:46.582 INFO:teuthology.orchestra.run.smithi178.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-02-21T23:38:46.582 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-21T23:38:46.582 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-21T23:38:46.582 INFO:teuthology.orchestra.run.smithi178.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-21T23:38:46.582 INFO:teuthology.orchestra.run.smithi178.stdout: python3-packaging-16.8-9.el8.noarch 2024-02-21T23:38:46.582 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-21T23:38:46.582 INFO:teuthology.orchestra.run.smithi178.stdout: python3-portend-2.6-1.el8.noarch 2024-02-21T23:38:46.582 INFO:teuthology.orchestra.run.smithi178.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-21T23:38:46.582 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-21T23:38:46.583 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-21T23:38:46.583 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-21T23:38:46.583 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-21T23:38:46.583 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-21T23:38:46.583 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rados-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:46.583 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rbd-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:46.583 INFO:teuthology.orchestra.run.smithi178.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-21T23:38:46.583 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-21T23:38:46.583 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rgw-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:46.583 INFO:teuthology.orchestra.run.smithi178.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-21T23:38:46.583 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-21T23:38:46.583 INFO:teuthology.orchestra.run.smithi178.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-02-21T23:38:46.583 INFO:teuthology.orchestra.run.smithi178.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-21T23:38:46.583 INFO:teuthology.orchestra.run.smithi178.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-21T23:38:46.584 INFO:teuthology.orchestra.run.smithi178.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-21T23:38:46.584 INFO:teuthology.orchestra.run.smithi178.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-21T23:38:46.584 INFO:teuthology.orchestra.run.smithi178.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-21T23:38:46.584 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-21T23:38:46.584 INFO:teuthology.orchestra.run.smithi178.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-21T23:38:46.584 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-21T23:38:46.584 INFO:teuthology.orchestra.run.smithi178.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-21T23:38:46.584 INFO:teuthology.orchestra.run.smithi178.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-21T23:38:46.584 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-fuse-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:46.584 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-mirror-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:46.584 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-nbd-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:38:46.584 INFO:teuthology.orchestra.run.smithi178.stdout: socat-1.7.4.1-1.el8.x86_64 2024-02-21T23:38:46.584 INFO:teuthology.orchestra.run.smithi178.stdout: thrift-0.13.0-2.el8.x86_64 2024-02-21T23:38:46.585 INFO:teuthology.orchestra.run.smithi178.stdout: uuid-1.6.2-43.el8.x86_64 2024-02-21T23:38:46.585 INFO:teuthology.orchestra.run.smithi178.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-02-21T23:38:46.585 INFO:teuthology.orchestra.run.smithi178.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-02-21T23:38:46.585 INFO:teuthology.orchestra.run.smithi178.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2024-02-21T23:38:46.585 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-21T23:38:46.585 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-02-21T23:38:46.802 DEBUG:teuthology.parallel:result is None 2024-02-21T23:39:03.748 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-selinux-2:18.2.1-638.gc1dc18f4.el8.x86_64 107/162 2024-02-21T23:39:03.749 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /sys 2024-02-21T23:39:03.749 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /proc 2024-02-21T23:39:03.749 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /dev 2024-02-21T23:39:03.749 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /run 2024-02-21T23:39:03.749 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /mnt 2024-02-21T23:39:03.749 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /var/tmp 2024-02-21T23:39:03.749 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /home 2024-02-21T23:39:03.749 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /tmp 2024-02-21T23:39:03.749 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /dev 2024-02-21T23:39:03.749 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:39:04.141 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-volume-2:18.2.1-638.gc1dc18f4.el8.noarch 108/162 2024-02-21T23:39:05.708 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-volume-2:18.2.1-638.gc1dc18f4.el8.noarch 108/162 2024-02-21T23:39:05.912 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-osd-2:18.2.1-638.gc1dc18f4.el8.x86_64 109/162 2024-02-21T23:39:06.133 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-osd-2:18.2.1-638.gc1dc18f4.el8.x86_64 109/162 2024-02-21T23:39:06.307 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mds-2:18.2.1-638.gc1dc18f4.el8.x86_64 110/162 2024-02-21T23:39:06.763 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mds-2:18.2.1-638.gc1dc18f4.el8.x86_64 110/162 2024-02-21T23:39:06.962 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mon-2:18.2.1-638.gc1dc18f4.el8.x86_64 111/162 2024-02-21T23:39:06.980 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mon-2:18.2.1-638.gc1dc18f4.el8.x86_64 111/162 2024-02-21T23:39:08.780 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: cephadm-2:18.2.1-638.gc1dc18f4.el8.noarch 112/162 2024-02-21T23:39:08.940 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : cephadm-2:18.2.1-638.gc1dc18f4.el8.noarch 112/162 2024-02-21T23:39:09.104 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-prometheus-alerts-2:18.2.1-638.gc1dc18f4.el 113/162 2024-02-21T23:39:09.283 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-grafana-dashboards-2:18.2.1-638.gc1dc18f4.e 114/162 2024-02-21T23:39:09.532 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-cephadm-2:18.2.1-638.gc1dc18f4.el8.noar 115/162 2024-02-21T23:39:09.756 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.1-638.gc1dc18f4.el8.noar 115/162 2024-02-21T23:39:10.195 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-dashboard-2:18.2.1-638.gc1dc18f4.el8.no 116/162 2024-02-21T23:39:11.157 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.1-638.gc1dc18f4.el8.no 116/162 2024-02-21T23:39:11.379 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.1-638.gc1dc 117/162 2024-02-21T23:39:11.473 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.1-638.gc1dc 117/162 2024-02-21T23:39:11.865 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-modules-core-2:18.2.1-638.gc1dc18f4.el8 118/162 2024-02-21T23:39:12.070 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-2:18.2.1-638.gc1dc18f4.el8.x86_64 119/162 2024-02-21T23:39:12.109 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-2:18.2.1-638.gc1dc18f4.el8.x86_64 119/162 2024-02-21T23:39:12.269 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-rook-2:18.2.1-638.gc1dc18f4.el8.noarch 120/162 2024-02-21T23:39:12.276 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.1-638.gc1dc18f4.el8.noarch 120/162 2024-02-21T23:39:13.569 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-2:18.2.1-638.gc1dc18f4.el8.x86_64 121/162 2024-02-21T23:39:13.853 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-radosgw-2:18.2.1-638.gc1dc18f4.el8.x86_64 122/162 2024-02-21T23:39:13.889 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-radosgw-2:18.2.1-638.gc1dc18f4.el8.x86_64 122/162 2024-02-21T23:39:14.090 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-immutable-object-cache-2:18.2.1-638.gc1dc18 123/162 2024-02-21T23:39:14.134 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.1-638.gc1dc18 123/162 2024-02-21T23:39:14.349 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : cephfs-mirror-2:18.2.1-638.gc1dc18f4.el8.x86_64 124/162 2024-02-21T23:39:14.694 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: cephfs-mirror-2:18.2.1-638.gc1dc18f4.el8.x86_64 124/162 2024-02-21T23:39:14.957 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-mirror-2:18.2.1-638.gc1dc18f4.el8.x86_64 125/162 2024-02-21T23:39:19.379 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-mirror-2:18.2.1-638.gc1dc18f4.el8.x86_64 125/162 2024-02-21T23:39:20.366 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-test-2:18.2.1-638.gc1dc18f4.el8.x86_64 126/162 2024-02-21T23:39:20.686 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openssl-devel-1:1.1.1k-12.el8_9.x86_64 127/162 2024-02-21T23:39:20.964 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 128/162 2024-02-21T23:39:21.371 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-fuse-2:18.2.1-638.gc1dc18f4.el8.x86_64 129/162 2024-02-21T23:39:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : postgresql-contrib-10.23-3.module+el8.9.0+20831+ 130/162 2024-02-21T23:39:21.914 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libtool-2.4.6-25.el8.x86_64 131/162 2024-02-21T23:39:21.928 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 131/162 2024-02-21T23:39:22.192 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 132/162 2024-02-21T23:39:22.385 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 133/162 2024-02-21T23:39:22.633 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephfs-devel-2:18.2.1-638.gc1dc18f4.el8.x86_6 134/162 2024-02-21T23:39:24.950 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: postgresql-server-10.23-3.module+el8.9.0+20831+7 135/162 2024-02-21T23:39:25.274 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : postgresql-server-10.23-3.module+el8.9.0+20831+7 135/162 2024-02-21T23:39:25.311 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: postgresql-server-10.23-3.module+el8.9.0+20831+7 135/162 2024-02-21T23:39:25.827 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 136/162 2024-02-21T23:39:26.280 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : iproute-6.2.0-5.el8_9.x86_64 137/162 2024-02-21T23:39:26.527 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : xfsdump-3.1.8-7.el8_9.x86_64 138/162 2024-02-21T23:39:26.748 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : cephfs-top-2:18.2.1-638.gc1dc18f4.el8.noarch 139/162 2024-02-21T23:39:27.040 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-fuse-2:18.2.1-638.gc1dc18f4.el8.x86_64 140/162 2024-02-21T23:39:27.402 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-nbd-2:18.2.1-638.gc1dc18f4.el8.x86_64 141/162 2024-02-21T23:39:27.742 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : openssl-1:1.1.1k-12.el8_9.x86_64 142/162 2024-02-21T23:39:27.939 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flex-2.6.1-9.el8.x86_64 143/162 2024-02-21T23:39:28.019 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 143/162 2024-02-21T23:39:28.231 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : bison-3.0.4-10.el8.x86_64 144/162 2024-02-21T23:39:28.254 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 144/162 2024-02-21T23:39:28.465 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : NetworkManager-initscripts-updown-1:1.40.16-13.e 145/162 2024-02-21T23:39:28.474 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-13.e 145/162 2024-02-21T23:39:28.600 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 146/162 2024-02-21T23:39:28.841 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : elfutils-libs-0.186-1.el8.x86_64 147/162 2024-02-21T23:39:29.025 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 148/162 2024-02-21T23:39:29.235 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 149/162 2024-02-21T23:39:29.235 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-02-21T23:39:29.551 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-02-21T23:39:29.551 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-02-21T23:39:29.801 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-02-21T23:39:29.802 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 152/162 2024-02-21T23:39:30.096 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 152/162 2024-02-21T23:39:30.358 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 153/162 2024-02-21T23:39:30.533 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : util-linux-user-2.32.1-35.el8.x86_64 154/162 2024-02-21T23:39:30.800 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : util-linux-2.32.1-35.el8.x86_64 155/162 2024-02-21T23:39:30.800 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : libfdisk-2.32.1-35.el8.x86_64 156/162 2024-02-21T23:39:31.084 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libfdisk-2.32.1-35.el8.x86_64 156/162 2024-02-21T23:39:31.084 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : libmount-2.32.1-35.el8.x86_64 157/162 2024-02-21T23:39:31.367 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libmount-2.32.1-35.el8.x86_64 157/162 2024-02-21T23:39:31.367 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : libblkid-2.32.1-35.el8.x86_64 158/162 2024-02-21T23:39:31.608 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libblkid-2.32.1-35.el8.x86_64 158/162 2024-02-21T23:39:31.608 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : libuuid-2.32.1-35.el8.x86_64 159/162 2024-02-21T23:39:31.875 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libuuid-2.32.1-35.el8.x86_64 159/162 2024-02-21T23:39:31.875 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-02-21T23:39:32.142 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-02-21T23:39:32.308 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : elfutils-libelf-0.186-1.el8.x86_64 161/162 2024-02-21T23:39:32.308 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-02-21T23:39:33.267 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-02-21T23:39:33.267 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-2:18.2.1-638.gc1dc18f4.el8.x86_64 1/162 2024-02-21T23:39:33.267 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-base-2:18.2.1-638.gc1dc18f4.el8.x86_64 2/162 2024-02-21T23:39:33.267 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-common-2:18.2.1-638.gc1dc18f4.el8.x86_64 3/162 2024-02-21T23:39:33.267 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-fuse-2:18.2.1-638.gc1dc18f4.el8.x86_64 4/162 2024-02-21T23:39:33.267 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-immutable-object-cache-2:18.2.1-638.gc1dc18 5/162 2024-02-21T23:39:33.268 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mds-2:18.2.1-638.gc1dc18f4.el8.x86_64 6/162 2024-02-21T23:39:33.268 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-2:18.2.1-638.gc1dc18f4.el8.x86_64 7/162 2024-02-21T23:39:33.268 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mon-2:18.2.1-638.gc1dc18f4.el8.x86_64 8/162 2024-02-21T23:39:33.268 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-osd-2:18.2.1-638.gc1dc18f4.el8.x86_64 9/162 2024-02-21T23:39:33.268 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-radosgw-2:18.2.1-638.gc1dc18f4.el8.x86_64 10/162 2024-02-21T23:39:33.268 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-selinux-2:18.2.1-638.gc1dc18f4.el8.x86_64 11/162 2024-02-21T23:39:33.268 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-test-2:18.2.1-638.gc1dc18f4.el8.x86_64 12/162 2024-02-21T23:39:33.268 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : cephfs-mirror-2:18.2.1-638.gc1dc18f4.el8.x86_64 13/162 2024-02-21T23:39:33.268 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephfs-devel-2:18.2.1-638.gc1dc18f4.el8.x86_6 14/162 2024-02-21T23:39:33.268 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephfs2-2:18.2.1-638.gc1dc18f4.el8.x86_64 15/162 2024-02-21T23:39:33.268 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephsqlite-2:18.2.1-638.gc1dc18f4.el8.x86_64 16/162 2024-02-21T23:39:33.268 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados-devel-2:18.2.1-638.gc1dc18f4.el8.x86_64 17/162 2024-02-21T23:39:33.269 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libradosstriper1-2:18.2.1-638.gc1dc18f4.el8.x86_ 18/162 2024-02-21T23:39:33.269 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librgw2-2:18.2.1-638.gc1dc18f4.el8.x86_64 19/162 2024-02-21T23:39:33.269 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ceph-argparse-2:18.2.1-638.gc1dc18f4.el8 20/162 2024-02-21T23:39:33.269 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ceph-common-2:18.2.1-638.gc1dc18f4.el8.x 21/162 2024-02-21T23:39:33.269 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cephfs-2:18.2.1-638.gc1dc18f4.el8.x86_64 22/162 2024-02-21T23:39:33.269 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rados-2:18.2.1-638.gc1dc18f4.el8.x86_64 23/162 2024-02-21T23:39:33.269 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rbd-2:18.2.1-638.gc1dc18f4.el8.x86_64 24/162 2024-02-21T23:39:33.269 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rgw-2:18.2.1-638.gc1dc18f4.el8.x86_64 25/162 2024-02-21T23:39:33.269 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-fuse-2:18.2.1-638.gc1dc18f4.el8.x86_64 26/162 2024-02-21T23:39:33.269 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-mirror-2:18.2.1-638.gc1dc18f4.el8.x86_64 27/162 2024-02-21T23:39:33.269 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-nbd-2:18.2.1-638.gc1dc18f4.el8.x86_64 28/162 2024-02-21T23:39:33.269 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-grafana-dashboards-2:18.2.1-638.gc1dc18f4.e 29/162 2024-02-21T23:39:33.269 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-cephadm-2:18.2.1-638.gc1dc18f4.el8.noar 30/162 2024-02-21T23:39:33.269 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-dashboard-2:18.2.1-638.gc1dc18f4.el8.no 31/162 2024-02-21T23:39:33.270 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.1-638.gc1dc 32/162 2024-02-21T23:39:33.270 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-modules-core-2:18.2.1-638.gc1dc18f4.el8 33/162 2024-02-21T23:39:33.270 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-rook-2:18.2.1-638.gc1dc18f4.el8.noarch 34/162 2024-02-21T23:39:33.270 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-prometheus-alerts-2:18.2.1-638.gc1dc18f4.el 35/162 2024-02-21T23:39:33.270 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-volume-2:18.2.1-638.gc1dc18f4.el8.noarch 36/162 2024-02-21T23:39:33.270 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : cephadm-2:18.2.1-638.gc1dc18f4.el8.noarch 37/162 2024-02-21T23:39:33.270 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : cephfs-top-2:18.2.1-638.gc1dc18f4.el8.noarch 38/162 2024-02-21T23:39:33.270 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/162 2024-02-21T23:39:33.270 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/162 2024-02-21T23:39:33.270 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/162 2024-02-21T23:39:33.270 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/162 2024-02-21T23:39:33.270 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/162 2024-02-21T23:39:33.270 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/162 2024-02-21T23:39:33.271 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/162 2024-02-21T23:39:33.271 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/162 2024-02-21T23:39:33.271 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/162 2024-02-21T23:39:33.271 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/162 2024-02-21T23:39:33.271 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/162 2024-02-21T23:39:33.271 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/162 2024-02-21T23:39:33.271 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/162 2024-02-21T23:39:33.271 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/162 2024-02-21T23:39:33.271 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/162 2024-02-21T23:39:33.271 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/162 2024-02-21T23:39:33.271 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/162 2024-02-21T23:39:33.271 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/162 2024-02-21T23:39:33.271 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/162 2024-02-21T23:39:33.271 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/162 2024-02-21T23:39:33.272 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/162 2024-02-21T23:39:33.272 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/162 2024-02-21T23:39:33.272 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/162 2024-02-21T23:39:33.272 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/162 2024-02-21T23:39:33.272 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/162 2024-02-21T23:39:33.272 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/162 2024-02-21T23:39:33.272 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/162 2024-02-21T23:39:33.272 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/162 2024-02-21T23:39:33.272 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/162 2024-02-21T23:39:33.272 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/162 2024-02-21T23:39:33.272 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/162 2024-02-21T23:39:33.272 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/162 2024-02-21T23:39:33.273 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/162 2024-02-21T23:39:33.273 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 72/162 2024-02-21T23:39:33.273 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 73/162 2024-02-21T23:39:33.273 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/162 2024-02-21T23:39:33.273 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 75/162 2024-02-21T23:39:33.273 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 76/162 2024-02-21T23:39:33.273 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 77/162 2024-02-21T23:39:33.273 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 78/162 2024-02-21T23:39:33.273 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 79/162 2024-02-21T23:39:33.273 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 80/162 2024-02-21T23:39:33.273 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 81/162 2024-02-21T23:39:33.273 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 82/162 2024-02-21T23:39:33.273 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 83/162 2024-02-21T23:39:33.273 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 84/162 2024-02-21T23:39:33.274 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 85/162 2024-02-21T23:39:33.274 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 86/162 2024-02-21T23:39:33.274 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 87/162 2024-02-21T23:39:33.274 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 88/162 2024-02-21T23:39:33.274 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 89/162 2024-02-21T23:39:33.274 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 90/162 2024-02-21T23:39:33.274 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 91/162 2024-02-21T23:39:33.274 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 92/162 2024-02-21T23:39:33.274 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 93/162 2024-02-21T23:39:33.274 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 94/162 2024-02-21T23:39:33.274 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-13.e 95/162 2024-02-21T23:39:33.274 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 96/162 2024-02-21T23:39:33.274 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : krb5-devel-1.18.2-26.el8_9.x86_64 97/162 2024-02-21T23:39:33.274 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 98/162 2024-02-21T23:39:33.274 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8_9.x86_64 99/162 2024-02-21T23:39:33.274 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 100/162 2024-02-21T23:39:33.275 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 101/162 2024-02-21T23:39:33.275 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 102/162 2024-02-21T23:39:33.275 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 103/162 2024-02-21T23:39:33.275 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 104/162 2024-02-21T23:39:33.275 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 105/162 2024-02-21T23:39:33.275 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : bison-3.0.4-10.el8.x86_64 106/162 2024-02-21T23:39:33.275 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flex-2.6.1-9.el8.x86_64 107/162 2024-02-21T23:39:33.275 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 108/162 2024-02-21T23:39:33.275 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 109/162 2024-02-21T23:39:33.275 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 110/162 2024-02-21T23:39:33.275 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 111/162 2024-02-21T23:39:33.275 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 112/162 2024-02-21T23:39:33.275 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 113/162 2024-02-21T23:39:33.275 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 114/162 2024-02-21T23:39:33.276 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 115/162 2024-02-21T23:39:33.276 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 116/162 2024-02-21T23:39:33.276 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 117/162 2024-02-21T23:39:33.276 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 118/162 2024-02-21T23:39:33.276 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 119/162 2024-02-21T23:39:33.276 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 120/162 2024-02-21T23:39:33.276 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 121/162 2024-02-21T23:39:33.276 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 122/162 2024-02-21T23:39:33.276 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : automake-1.16.1-8.el8.noarch 123/162 2024-02-21T23:39:33.276 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libpq-13.11-1.el8.x86_64 124/162 2024-02-21T23:39:33.277 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : jq-1.6-7.el8.x86_64 125/162 2024-02-21T23:39:33.277 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 126/162 2024-02-21T23:39:33.277 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : postgresql-10.23-3.module+el8.9.0+20831+78489779 127/162 2024-02-21T23:39:33.277 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : postgresql-contrib-10.23-3.module+el8.9.0+20831+ 128/162 2024-02-21T23:39:33.277 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : postgresql-server-10.23-3.module+el8.9.0+20831+7 129/162 2024-02-21T23:39:33.277 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 130/162 2024-02-21T23:39:33.277 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados2-2:18.2.1-638.gc1dc18f4.el8.x86_64 131/162 2024-02-21T23:39:33.277 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 132/162 2024-02-21T23:39:33.277 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librbd1-2:18.2.1-638.gc1dc18f4.el8.x86_64 133/162 2024-02-21T23:39:33.277 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 134/162 2024-02-21T23:39:33.277 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 135/162 2024-02-21T23:39:33.277 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : elfutils-libelf-0.186-1.el8.x86_64 136/162 2024-02-21T23:39:33.277 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 137/162 2024-02-21T23:39:33.277 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : elfutils-libs-0.186-1.el8.x86_64 138/162 2024-02-21T23:39:33.277 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 139/162 2024-02-21T23:39:33.278 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libmount-2.32.1-35.el8.x86_64 140/162 2024-02-21T23:39:33.278 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 141/162 2024-02-21T23:39:33.278 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libsmartcols-2.32.1-35.el8.x86_64 142/162 2024-02-21T23:39:33.278 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 143/162 2024-02-21T23:39:33.278 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libuuid-2.32.1-35.el8.x86_64 144/162 2024-02-21T23:39:33.278 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 145/162 2024-02-21T23:39:33.278 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : util-linux-2.32.1-35.el8.x86_64 146/162 2024-02-21T23:39:33.278 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 147/162 2024-02-21T23:39:33.278 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : util-linux-user-2.32.1-35.el8.x86_64 148/162 2024-02-21T23:39:33.278 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : iproute-6.2.0-5.el8_9.x86_64 149/162 2024-02-21T23:39:33.278 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 150/162 2024-02-21T23:39:33.278 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 151/162 2024-02-21T23:39:33.278 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libblkid-2.32.1-35.el8.x86_64 152/162 2024-02-21T23:39:33.278 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 153/162 2024-02-21T23:39:33.278 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libfdisk-2.32.1-35.el8.x86_64 154/162 2024-02-21T23:39:33.279 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openssl-1:1.1.1k-12.el8_9.x86_64 155/162 2024-02-21T23:39:33.279 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 156/162 2024-02-21T23:39:33.279 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8_9.x86_64 157/162 2024-02-21T23:39:33.279 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 158/162 2024-02-21T23:39:33.279 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : xfsdump-3.1.8-7.el8_9.x86_64 159/162 2024-02-21T23:39:33.279 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 160/162 2024-02-21T23:39:33.279 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 161/162 2024-02-21T23:39:43.608 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 162/162 2024-02-21T23:39:43.609 INFO:teuthology.orchestra.run.smithi090.stdout:Installed products updated. 2024-02-21T23:39:44.237 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:39:44.237 INFO:teuthology.orchestra.run.smithi090.stdout:Upgraded: 2024-02-21T23:39:44.237 INFO:teuthology.orchestra.run.smithi090.stdout: elfutils-libelf-0.189-3.el8.x86_64 2024-02-21T23:39:44.237 INFO:teuthology.orchestra.run.smithi090.stdout: elfutils-libs-0.189-3.el8.x86_64 2024-02-21T23:39:44.237 INFO:teuthology.orchestra.run.smithi090.stdout: iproute-6.2.0-5.el8_9.x86_64 2024-02-21T23:39:44.237 INFO:teuthology.orchestra.run.smithi090.stdout: libblkid-2.32.1-43.el8.x86_64 2024-02-21T23:39:44.238 INFO:teuthology.orchestra.run.smithi090.stdout: libfdisk-2.32.1-43.el8.x86_64 2024-02-21T23:39:44.238 INFO:teuthology.orchestra.run.smithi090.stdout: libmount-2.32.1-43.el8.x86_64 2024-02-21T23:39:44.238 INFO:teuthology.orchestra.run.smithi090.stdout: librados2-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:39:44.238 INFO:teuthology.orchestra.run.smithi090.stdout: librbd1-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:39:44.238 INFO:teuthology.orchestra.run.smithi090.stdout: libsmartcols-2.32.1-43.el8.x86_64 2024-02-21T23:39:44.238 INFO:teuthology.orchestra.run.smithi090.stdout: libuuid-2.32.1-43.el8.x86_64 2024-02-21T23:39:44.238 INFO:teuthology.orchestra.run.smithi090.stdout: openssl-1:1.1.1k-12.el8_9.x86_64 2024-02-21T23:39:44.238 INFO:teuthology.orchestra.run.smithi090.stdout: openssl-libs-1:1.1.1k-12.el8_9.x86_64 2024-02-21T23:39:44.238 INFO:teuthology.orchestra.run.smithi090.stdout: util-linux-2.32.1-43.el8.x86_64 2024-02-21T23:39:44.238 INFO:teuthology.orchestra.run.smithi090.stdout: util-linux-user-2.32.1-43.el8.x86_64 2024-02-21T23:39:44.238 INFO:teuthology.orchestra.run.smithi090.stdout: xfsdump-3.1.8-7.el8_9.x86_64 2024-02-21T23:39:44.238 INFO:teuthology.orchestra.run.smithi090.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-02-21T23:39:44.238 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-21T23:39:44.239 INFO:teuthology.orchestra.run.smithi090.stdout: NetworkManager-initscripts-updown-1:1.40.16-13.el8_9.noarch 2024-02-21T23:39:44.239 INFO:teuthology.orchestra.run.smithi090.stdout: automake-1.16.1-8.el8.noarch 2024-02-21T23:39:44.239 INFO:teuthology.orchestra.run.smithi090.stdout: bison-3.0.4-10.el8.x86_64 2024-02-21T23:39:44.239 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:39:44.239 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:39:44.239 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-common-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:39:44.239 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-fuse-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:39:44.239 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-grafana-dashboards-2:18.2.1-638.gc1dc18f4.el8.noarch 2024-02-21T23:39:44.239 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-immutable-object-cache-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:39:44.239 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mds-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:39:44.239 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:39:44.239 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-cephadm-2:18.2.1-638.gc1dc18f4.el8.noarch 2024-02-21T23:39:44.239 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-dashboard-2:18.2.1-638.gc1dc18f4.el8.noarch 2024-02-21T23:39:44.239 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-diskprediction-local-2:18.2.1-638.gc1dc18f4.el8.noarch 2024-02-21T23:39:44.240 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-modules-core-2:18.2.1-638.gc1dc18f4.el8.noarch 2024-02-21T23:39:44.240 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-rook-2:18.2.1-638.gc1dc18f4.el8.noarch 2024-02-21T23:39:44.240 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mon-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:39:44.240 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-osd-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:39:44.240 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-prometheus-alerts-2:18.2.1-638.gc1dc18f4.el8.noarch 2024-02-21T23:39:44.240 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-radosgw-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:39:44.240 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-selinux-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:39:44.240 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-test-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:39:44.240 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-volume-2:18.2.1-638.gc1dc18f4.el8.noarch 2024-02-21T23:39:44.240 INFO:teuthology.orchestra.run.smithi090.stdout: cephadm-2:18.2.1-638.gc1dc18f4.el8.noarch 2024-02-21T23:39:44.240 INFO:teuthology.orchestra.run.smithi090.stdout: cephfs-mirror-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:39:44.240 INFO:teuthology.orchestra.run.smithi090.stdout: cephfs-top-2:18.2.1-638.gc1dc18f4.el8.noarch 2024-02-21T23:39:44.241 INFO:teuthology.orchestra.run.smithi090.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2024-02-21T23:39:44.241 INFO:teuthology.orchestra.run.smithi090.stdout: flex-2.6.1-9.el8.x86_64 2024-02-21T23:39:44.241 INFO:teuthology.orchestra.run.smithi090.stdout: fuse-2.9.7-17.el8.x86_64 2024-02-21T23:39:44.241 INFO:teuthology.orchestra.run.smithi090.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-21T23:39:44.241 INFO:teuthology.orchestra.run.smithi090.stdout: jq-1.6-7.el8.x86_64 2024-02-21T23:39:44.241 INFO:teuthology.orchestra.run.smithi090.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-02-21T23:39:44.241 INFO:teuthology.orchestra.run.smithi090.stdout: krb5-devel-1.18.2-26.el8_9.x86_64 2024-02-21T23:39:44.241 INFO:teuthology.orchestra.run.smithi090.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-02-21T23:39:44.241 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs-devel-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:39:44.241 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs2-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:39:44.241 INFO:teuthology.orchestra.run.smithi090.stdout: libcephsqlite-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:39:44.241 INFO:teuthology.orchestra.run.smithi090.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2024-02-21T23:39:44.241 INFO:teuthology.orchestra.run.smithi090.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-02-21T23:39:44.241 INFO:teuthology.orchestra.run.smithi090.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-21T23:39:44.242 INFO:teuthology.orchestra.run.smithi090.stdout: libpq-13.11-1.el8.x86_64 2024-02-21T23:39:44.242 INFO:teuthology.orchestra.run.smithi090.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-02-21T23:39:44.242 INFO:teuthology.orchestra.run.smithi090.stdout: librabbitmq-0.9.0-4.el8.x86_64 2024-02-21T23:39:44.242 INFO:teuthology.orchestra.run.smithi090.stdout: librados-devel-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:39:44.242 INFO:teuthology.orchestra.run.smithi090.stdout: libradosstriper1-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:39:44.242 INFO:teuthology.orchestra.run.smithi090.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-02-21T23:39:44.242 INFO:teuthology.orchestra.run.smithi090.stdout: librgw2-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:39:44.242 INFO:teuthology.orchestra.run.smithi090.stdout: libselinux-devel-2.9-5.el8.x86_64 2024-02-21T23:39:44.242 INFO:teuthology.orchestra.run.smithi090.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-02-21T23:39:44.242 INFO:teuthology.orchestra.run.smithi090.stdout: libtool-2.4.6-25.el8.x86_64 2024-02-21T23:39:44.242 INFO:teuthology.orchestra.run.smithi090.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-21T23:39:44.242 INFO:teuthology.orchestra.run.smithi090.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2024-02-21T23:39:44.242 INFO:teuthology.orchestra.run.smithi090.stdout: libverto-devel-0.3.0-5.el8.x86_64 2024-02-21T23:39:44.243 INFO:teuthology.orchestra.run.smithi090.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-02-21T23:39:44.243 INFO:teuthology.orchestra.run.smithi090.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-21T23:39:44.243 INFO:teuthology.orchestra.run.smithi090.stdout: oniguruma-6.8.2-2.el8.x86_64 2024-02-21T23:39:44.243 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-0.3.15-6.el8.x86_64 2024-02-21T23:39:44.243 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-02-21T23:39:44.243 INFO:teuthology.orchestra.run.smithi090.stdout: openssl-devel-1:1.1.1k-12.el8_9.x86_64 2024-02-21T23:39:44.243 INFO:teuthology.orchestra.run.smithi090.stdout: pcre2-devel-10.32-2.el8.x86_64 2024-02-21T23:39:44.243 INFO:teuthology.orchestra.run.smithi090.stdout: pcre2-utf16-10.32-2.el8.x86_64 2024-02-21T23:39:44.243 INFO:teuthology.orchestra.run.smithi090.stdout: pcre2-utf32-10.32-2.el8.x86_64 2024-02-21T23:39:44.243 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-02-21T23:39:44.243 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-contrib-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-02-21T23:39:44.243 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-server-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-02-21T23:39:44.243 INFO:teuthology.orchestra.run.smithi090.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-02-21T23:39:44.243 INFO:teuthology.orchestra.run.smithi090.stdout: python3-babel-2.5.1-7.el8.noarch 2024-02-21T23:39:44.244 INFO:teuthology.orchestra.run.smithi090.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-21T23:39:44.244 INFO:teuthology.orchestra.run.smithi090.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-21T23:39:44.244 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-21T23:39:44.244 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-argparse-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:39:44.244 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-common-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:39:44.244 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cephfs-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:39:44.244 INFO:teuthology.orchestra.run.smithi090.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-21T23:39:44.244 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-21T23:39:44.244 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-21T23:39:44.244 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-21T23:39:44.244 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2024-02-21T23:39:44.244 INFO:teuthology.orchestra.run.smithi090.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-21T23:39:44.244 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-21T23:39:44.245 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-21T23:39:44.245 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-02-21T23:39:44.245 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-21T23:39:44.245 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-21T23:39:44.245 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-21T23:39:44.245 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-21T23:39:44.245 INFO:teuthology.orchestra.run.smithi090.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-21T23:39:44.245 INFO:teuthology.orchestra.run.smithi090.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-21T23:39:44.245 INFO:teuthology.orchestra.run.smithi090.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-21T23:39:44.245 INFO:teuthology.orchestra.run.smithi090.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-21T23:39:44.245 INFO:teuthology.orchestra.run.smithi090.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-02-21T23:39:44.245 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-21T23:39:44.245 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-21T23:39:44.245 INFO:teuthology.orchestra.run.smithi090.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-21T23:39:44.246 INFO:teuthology.orchestra.run.smithi090.stdout: python3-packaging-16.8-9.el8.noarch 2024-02-21T23:39:44.246 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-21T23:39:44.246 INFO:teuthology.orchestra.run.smithi090.stdout: python3-portend-2.6-1.el8.noarch 2024-02-21T23:39:44.246 INFO:teuthology.orchestra.run.smithi090.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-21T23:39:44.246 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-21T23:39:44.246 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-21T23:39:44.246 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-21T23:39:44.246 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-21T23:39:44.246 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-21T23:39:44.246 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:39:44.246 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rbd-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:39:44.246 INFO:teuthology.orchestra.run.smithi090.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-21T23:39:44.246 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-21T23:39:44.247 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:39:44.247 INFO:teuthology.orchestra.run.smithi090.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-21T23:39:44.247 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-21T23:39:44.247 INFO:teuthology.orchestra.run.smithi090.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-02-21T23:39:44.247 INFO:teuthology.orchestra.run.smithi090.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-21T23:39:44.247 INFO:teuthology.orchestra.run.smithi090.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-21T23:39:44.247 INFO:teuthology.orchestra.run.smithi090.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-21T23:39:44.247 INFO:teuthology.orchestra.run.smithi090.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-21T23:39:44.247 INFO:teuthology.orchestra.run.smithi090.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-21T23:39:44.247 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-21T23:39:44.247 INFO:teuthology.orchestra.run.smithi090.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-21T23:39:44.247 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-21T23:39:44.247 INFO:teuthology.orchestra.run.smithi090.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-21T23:39:44.247 INFO:teuthology.orchestra.run.smithi090.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-21T23:39:44.248 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-fuse-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:39:44.248 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-mirror-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:39:44.248 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-nbd-2:18.2.1-638.gc1dc18f4.el8.x86_64 2024-02-21T23:39:44.248 INFO:teuthology.orchestra.run.smithi090.stdout: socat-1.7.4.1-1.el8.x86_64 2024-02-21T23:39:44.248 INFO:teuthology.orchestra.run.smithi090.stdout: thrift-0.13.0-2.el8.x86_64 2024-02-21T23:39:44.248 INFO:teuthology.orchestra.run.smithi090.stdout: uuid-1.6.2-43.el8.x86_64 2024-02-21T23:39:44.248 INFO:teuthology.orchestra.run.smithi090.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-02-21T23:39:44.248 INFO:teuthology.orchestra.run.smithi090.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-02-21T23:39:44.249 INFO:teuthology.orchestra.run.smithi090.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2024-02-21T23:39:44.249 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:39:44.249 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-21T23:39:44.459 DEBUG:teuthology.parallel:result is None 2024-02-21T23:39:44.460 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=c1dc18f444603397768fcb7c34a8488b10284df6 2024-02-21T23:39:44.679 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-21T23:39:44.703 INFO:teuthology.orchestra.run.smithi090.stdout:18.2.1-638.gc1dc18f4.el8 2024-02-21T23:39:44.704 INFO:teuthology.packaging:The installed version of ceph is 18.2.1-638.gc1dc18f4.el8 2024-02-21T23:39:44.704 INFO:teuthology.task.install:The correct ceph version 18.2.1-638.gc1dc18f4 is installed. 2024-02-21T23:39:44.706 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=c1dc18f444603397768fcb7c34a8488b10284df6 2024-02-21T23:39:44.849 DEBUG:teuthology.orchestra.run.smithi176:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-21T23:39:44.878 INFO:teuthology.orchestra.run.smithi176.stdout:18.2.1-638.gc1dc18f4.el8 2024-02-21T23:39:44.879 INFO:teuthology.packaging:The installed version of ceph is 18.2.1-638.gc1dc18f4.el8 2024-02-21T23:39:44.879 INFO:teuthology.task.install:The correct ceph version 18.2.1-638.gc1dc18f4 is installed. 2024-02-21T23:39:44.881 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=c1dc18f444603397768fcb7c34a8488b10284df6 2024-02-21T23:39:45.097 DEBUG:teuthology.orchestra.run.smithi178:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-21T23:39:45.127 INFO:teuthology.orchestra.run.smithi178.stdout:18.2.1-638.gc1dc18f4.el8 2024-02-21T23:39:45.127 INFO:teuthology.packaging:The installed version of ceph is 18.2.1-638.gc1dc18f4.el8 2024-02-21T23:39:45.127 INFO:teuthology.task.install:The correct ceph version 18.2.1-638.gc1dc18f4 is installed. 2024-02-21T23:39:45.129 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-21T23:39:45.129 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-21T23:39:45.130 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-21T23:39:45.176 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-02-21T23:39:45.176 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-21T23:39:45.225 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-02-21T23:39:45.225 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-21T23:39:45.273 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-21T23:39:45.273 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-21T23:39:45.273 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/daemon-helper 2024-02-21T23:39:45.304 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-21T23:39:45.373 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-02-21T23:39:45.373 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/daemon-helper 2024-02-21T23:39:45.403 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-21T23:39:45.474 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-02-21T23:39:45.474 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/daemon-helper 2024-02-21T23:39:45.507 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-21T23:39:45.578 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-21T23:39:45.578 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-21T23:39:45.579 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-21T23:39:45.610 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-21T23:39:45.680 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-02-21T23:39:45.680 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-21T23:39:45.711 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-21T23:39:45.781 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-02-21T23:39:45.781 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-21T23:39:45.813 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-21T23:39:45.884 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-21T23:39:45.884 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-21T23:39:45.884 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/stdin-killer 2024-02-21T23:39:45.917 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-21T23:39:45.986 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-02-21T23:39:45.986 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/stdin-killer 2024-02-21T23:39:46.015 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-21T23:39:46.085 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-02-21T23:39:46.085 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/stdin-killer 2024-02-21T23:39:46.116 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-21T23:39:46.188 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-21T23:39:46.276 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 1, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': 'c1dc18f444603397768fcb7c34a8488b10284df6', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-02-21T23:39:46.276 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 2024-02-21T23:39:46.277 INFO:tasks.cephadm:Cluster fsid is 7ed81218-d112-11ee-95bd-87774f69a715 2024-02-21T23:39:46.277 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-21T23:39:46.277 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.90', 'mon.b': '172.21.15.176', 'mon.c': '172.21.15.178'} 2024-02-21T23:39:46.277 INFO:tasks.cephadm:First mon is mon.a on smithi090 2024-02-21T23:39:46.277 INFO:tasks.cephadm:First mgr is x 2024-02-21T23:39:46.277 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-21T23:39:46.278 DEBUG:teuthology.orchestra.run.smithi090:> sudo hostname $(hostname -s) 2024-02-21T23:39:46.310 DEBUG:teuthology.orchestra.run.smithi176:> sudo hostname $(hostname -s) 2024-02-21T23:39:46.341 DEBUG:teuthology.orchestra.run.smithi178:> sudo hostname $(hostname -s) 2024-02-21T23:39:46.373 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-02-21T23:39:46.374 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=c1dc18f444603397768fcb7c34a8488b10284df6 2024-02-21T23:39:46.516 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'c1dc18f444603397768fcb7c34a8488b10284df6', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/76981/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.2.1-638-gc1dc18f4', 'node_name': '172.21.2.8+braggi08', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.2.1-638.gc1dc18f4'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-02-21-1249-reef/c1dc18f444603397768fcb7c34a8488b10284df6/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2024-02-21 22:24:40.188410', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-yuri10-testing-2024-02-21-1249-reef', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-yuri10-testing-2024-02-21-1249-reef/c1dc18f444603397768fcb7c34a8488b10284df6/centos/8/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-02-21T23:39:46.653 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-yuri10-testing-2024-02-21-1249-reef, sha1 c1dc18f444603397768fcb7c34a8488b10284df6 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=c1dc18f444603397768fcb7c34a8488b10284df6 2024-02-21T23:39:46.654 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-yuri10-testing-2024-02-21-1249-reef/c1dc18f444603397768fcb7c34a8488b10284df6/centos/8/x86_64/flavors/default/cephadm 2024-02-21T23:39:46.655 DEBUG:teuthology.orchestra.run.smithi090:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-yuri10-testing-2024-02-21-1249-reef/c1dc18f444603397768fcb7c34a8488b10284df6/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-21T23:39:47.294 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-rw-r--. 1 ubuntu ubuntu 701072 Feb 21 23:39 /home/ubuntu/cephtest/cephadm 2024-02-21T23:39:47.295 DEBUG:teuthology.orchestra.run.smithi176:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-yuri10-testing-2024-02-21-1249-reef/c1dc18f444603397768fcb7c34a8488b10284df6/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-21T23:39:47.836 INFO:teuthology.orchestra.run.smithi176.stdout:-rw-rw-r--. 1 ubuntu ubuntu 701072 Feb 21 23:39 /home/ubuntu/cephtest/cephadm 2024-02-21T23:39:47.836 DEBUG:teuthology.orchestra.run.smithi178:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-yuri10-testing-2024-02-21-1249-reef/c1dc18f444603397768fcb7c34a8488b10284df6/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-21T23:39:48.406 INFO:teuthology.orchestra.run.smithi178.stdout:-rw-rw-r--. 1 ubuntu ubuntu 701072 Feb 21 23:39 /home/ubuntu/cephtest/cephadm 2024-02-21T23:39:48.406 DEBUG:teuthology.orchestra.run.smithi090:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-21T23:39:48.428 DEBUG:teuthology.orchestra.run.smithi176:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-21T23:39:48.449 DEBUG:teuthology.orchestra.run.smithi178:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-21T23:39:48.486 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 on all hosts... 2024-02-21T23:39:48.487 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 pull 2024-02-21T23:39:48.489 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 pull 2024-02-21T23:39:48.491 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 pull 2024-02-21T23:39:49.286 INFO:teuthology.orchestra.run.smithi178.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6... 2024-02-21T23:39:49.324 INFO:teuthology.orchestra.run.smithi176.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6... 2024-02-21T23:39:51.394 INFO:teuthology.orchestra.run.smithi090.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6... 2024-02-21T23:40:20.028 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-21T23:40:20.029 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_version": "ceph version 18.2.1-638-gc1dc18f4 (c1dc18f444603397768fcb7c34a8488b10284df6) reef (stable)", 2024-02-21T23:40:20.029 INFO:teuthology.orchestra.run.smithi090.stdout: "image_id": "f643fdc7b9a3ab61493be505fd5af5ebf7b55b8767c45a7f2091a0ac1353d09e", 2024-02-21T23:40:20.029 INFO:teuthology.orchestra.run.smithi090.stdout: "repo_digests": [ 2024-02-21T23:40:20.029 INFO:teuthology.orchestra.run.smithi090.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:75ec18f7067b26b32c6f90cd6827cc96a276ce86be68e998f0bd36838cbd5bd0" 2024-02-21T23:40:20.029 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-21T23:40:20.029 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-21T23:40:35.216 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-02-21T23:40:35.217 INFO:teuthology.orchestra.run.smithi176.stdout: "ceph_version": "ceph version 18.2.1-638-gc1dc18f4 (c1dc18f444603397768fcb7c34a8488b10284df6) reef (stable)", 2024-02-21T23:40:35.217 INFO:teuthology.orchestra.run.smithi176.stdout: "image_id": "f643fdc7b9a3ab61493be505fd5af5ebf7b55b8767c45a7f2091a0ac1353d09e", 2024-02-21T23:40:35.217 INFO:teuthology.orchestra.run.smithi176.stdout: "repo_digests": [ 2024-02-21T23:40:35.217 INFO:teuthology.orchestra.run.smithi176.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:75ec18f7067b26b32c6f90cd6827cc96a276ce86be68e998f0bd36838cbd5bd0" 2024-02-21T23:40:35.217 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-02-21T23:40:35.217 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-02-21T23:40:35.357 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-02-21T23:40:35.357 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_version": "ceph version 18.2.1-638-gc1dc18f4 (c1dc18f444603397768fcb7c34a8488b10284df6) reef (stable)", 2024-02-21T23:40:35.357 INFO:teuthology.orchestra.run.smithi178.stdout: "image_id": "f643fdc7b9a3ab61493be505fd5af5ebf7b55b8767c45a7f2091a0ac1353d09e", 2024-02-21T23:40:35.357 INFO:teuthology.orchestra.run.smithi178.stdout: "repo_digests": [ 2024-02-21T23:40:35.357 INFO:teuthology.orchestra.run.smithi178.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:75ec18f7067b26b32c6f90cd6827cc96a276ce86be68e998f0bd36838cbd5bd0" 2024-02-21T23:40:35.357 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-02-21T23:40:35.357 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-02-21T23:40:35.378 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /etc/ceph 2024-02-21T23:40:35.416 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /etc/ceph 2024-02-21T23:40:35.447 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /etc/ceph 2024-02-21T23:40:35.478 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 777 /etc/ceph 2024-02-21T23:40:35.510 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 777 /etc/ceph 2024-02-21T23:40:35.537 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 777 /etc/ceph 2024-02-21T23:40:35.564 INFO:tasks.cephadm:Writing seed config... 2024-02-21T23:40:35.565 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-02-21T23:40:35.565 INFO:tasks.cephadm: override: [client] debug client = 20 2024-02-21T23:40:35.565 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-02-21T23:40:35.565 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-02-21T23:40:35.566 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-02-21T23:40:35.566 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-02-21T23:40:35.566 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-02-21T23:40:35.566 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-02-21T23:40:35.566 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-02-21T23:40:35.566 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-02-21T23:40:35.566 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-02-21T23:40:35.566 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-02-21T23:40:35.566 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-02-21T23:40:35.566 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-02-21T23:40:35.566 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-02-21T23:40:35.566 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-02-21T23:40:35.566 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-02-21T23:40:35.567 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-02-21T23:40:35.567 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-02-21T23:40:35.567 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-02-21T23:40:35.567 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-02-21T23:40:35.567 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-02-21T23:40:35.567 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-21T23:40:35.567 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-21T23:40:35.567 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-21T23:40:35.567 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-21T23:40:35.567 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-21T23:40:35.567 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-02-21T23:40:35.567 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-02-21T23:40:35.567 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-02-21T23:40:35.567 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-02-21T23:40:35.568 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-02-21T23:40:35.568 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-02-21T23:40:35.568 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-21T23:40:35.568 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-21T23:40:35.568 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-02-21T23:40:35.568 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-02-21T23:40:35.568 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-02-21T23:40:35.568 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-02-21T23:40:35.568 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-02-21T23:40:35.568 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-02-21T23:40:35.568 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-02-21T23:40:35.568 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-02-21T23:40:35.568 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-02-21T23:40:35.569 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-21T23:40:35.569 DEBUG:teuthology.orchestra.run.smithi090:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-21T23:40:35.588 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 7ed81218-d112-11ee-95bd-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-02-21T23:40:35.589 DEBUG:teuthology.orchestra.run.smithi090:mon.a> sudo journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@mon.a.service 2024-02-21T23:40:35.631 DEBUG:teuthology.orchestra.run.smithi090:mgr.x> sudo journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@mgr.x.service 2024-02-21T23:40:35.660 INFO:journalctl@ceph.mon.a.smithi090.stdout:-- Logs begin at Wed 2024-02-21 23:24:49 UTC. -- 2024-02-21T23:40:35.661 INFO:tasks.cephadm:Bootstrapping... 2024-02-21T23:40:35.661 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 -v bootstrap --fsid 7ed81218-d112-11ee-95bd-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.90 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-21T23:40:35.688 INFO:journalctl@ceph.mgr.x.smithi090.stdout:-- Logs begin at Wed 2024-02-21 23:24:49 UTC. -- 2024-02-21T23:40:35.823 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-21T23:40:35.823 INFO:teuthology.orchestra.run.smithi090.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6', '-v', 'bootstrap', '--fsid', '7ed81218-d112-11ee-95bd-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.90', '--skip-admin-label'] 2024-02-21T23:40:35.858 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stdout 4.6.1 2024-02-21T23:40:35.858 INFO:teuthology.orchestra.run.smithi090.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-02-21T23:40:35.859 INFO:teuthology.orchestra.run.smithi090.stdout:Verifying podman|docker is present... 2024-02-21T23:40:35.895 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stdout 4.6.1 2024-02-21T23:40:35.895 INFO:teuthology.orchestra.run.smithi090.stdout:Verifying lvm2 is present... 2024-02-21T23:40:35.895 INFO:teuthology.orchestra.run.smithi090.stdout:Verifying time synchronization is in place... 2024-02-21T23:40:35.902 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-21T23:40:35.902 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-21T23:40:35.908 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-21T23:40:35.908 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout inactive 2024-02-21T23:40:35.914 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout enabled 2024-02-21T23:40:35.920 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout active 2024-02-21T23:40:35.920 INFO:teuthology.orchestra.run.smithi090.stdout:Unit chronyd.service is enabled and running 2024-02-21T23:40:35.920 INFO:teuthology.orchestra.run.smithi090.stdout:Repeating the final host check... 2024-02-21T23:40:35.951 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stdout 4.6.1 2024-02-21T23:40:35.951 INFO:teuthology.orchestra.run.smithi090.stdout:podman (/bin/podman) version 4.6.1 is present 2024-02-21T23:40:35.952 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl is present 2024-02-21T23:40:35.952 INFO:teuthology.orchestra.run.smithi090.stdout:lvcreate is present 2024-02-21T23:40:35.957 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-21T23:40:35.958 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-21T23:40:35.963 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-21T23:40:35.963 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout inactive 2024-02-21T23:40:35.969 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout enabled 2024-02-21T23:40:35.975 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout active 2024-02-21T23:40:35.975 INFO:teuthology.orchestra.run.smithi090.stdout:Unit chronyd.service is enabled and running 2024-02-21T23:40:35.975 INFO:teuthology.orchestra.run.smithi090.stdout:Host looks OK 2024-02-21T23:40:35.975 INFO:teuthology.orchestra.run.smithi090.stdout:Cluster fsid: 7ed81218-d112-11ee-95bd-87774f69a715 2024-02-21T23:40:35.975 INFO:teuthology.orchestra.run.smithi090.stdout:Acquiring lock 140484349762024 on /run/cephadm/7ed81218-d112-11ee-95bd-87774f69a715.lock 2024-02-21T23:40:35.975 INFO:teuthology.orchestra.run.smithi090.stdout:Lock 140484349762024 acquired on /run/cephadm/7ed81218-d112-11ee-95bd-87774f69a715.lock 2024-02-21T23:40:35.976 INFO:teuthology.orchestra.run.smithi090.stdout:Verifying IP 172.21.15.90 port 3300 ... 2024-02-21T23:40:35.976 INFO:teuthology.orchestra.run.smithi090.stdout:Verifying IP 172.21.15.90 port 6789 ... 2024-02-21T23:40:35.976 INFO:teuthology.orchestra.run.smithi090.stdout:Base mon IP(s) is [172.21.15.90:3300, 172.21.15.90:6789], mon addrv is [v2:172.21.15.90:3300,v1:172.21.15.90:6789] 2024-02-21T23:40:35.980 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp metric 100 2024-02-21T23:40:35.980 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.90 metric 100 2024-02-21T23:40:35.983 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-02-21T23:40:35.983 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-02-21T23:40:35.983 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-02-21T23:40:35.985 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-21T23:40:35.985 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-02-21T23:40:35.985 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-21T23:40:35.985 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-02-21T23:40:35.986 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout inet6 fe80::b06a:4dab:9131:21e7/64 scope link noprefixroute 2024-02-21T23:40:35.986 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-21T23:40:35.986 INFO:teuthology.orchestra.run.smithi090.stdout:Mon IP `172.21.15.90` is in CIDR network `172.21.0.0/20` 2024-02-21T23:40:35.986 INFO:teuthology.orchestra.run.smithi090.stdout:Mon IP `172.21.15.90` is in CIDR network `172.21.0.0/20` 2024-02-21T23:40:35.986 INFO:teuthology.orchestra.run.smithi090.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-02-21T23:40:35.987 INFO:teuthology.orchestra.run.smithi090.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-21T23:40:35.988 INFO:teuthology.orchestra.run.smithi090.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6... 2024-02-21T23:40:36.377 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stdout f643fdc7b9a3ab61493be505fd5af5ebf7b55b8767c45a7f2091a0ac1353d09e 2024-02-21T23:40:36.378 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6... 2024-02-21T23:40:36.378 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stderr Getting image source signatures 2024-02-21T23:40:36.378 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stderr Copying blob sha256:98cd903a9d0d54e90ba9a83e992df9e8d083486541b9a308de90ea6902b85e03 2024-02-21T23:40:36.378 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stderr Copying blob sha256:0f642e61a11afb42c0ccb5b05f5809250ad643ed143780d6ff49a5a4c492fc38 2024-02-21T23:40:36.378 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stderr Copying config sha256:f643fdc7b9a3ab61493be505fd5af5ebf7b55b8767c45a7f2091a0ac1353d09e 2024-02-21T23:40:36.378 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stderr Writing manifest to image destination 2024-02-21T23:40:37.084 INFO:teuthology.orchestra.run.smithi090.stdout:ceph: stdout ceph version 18.2.1-638-gc1dc18f4 (c1dc18f444603397768fcb7c34a8488b10284df6) reef (stable) 2024-02-21T23:40:37.084 INFO:teuthology.orchestra.run.smithi090.stdout:Ceph version: ceph version 18.2.1-638-gc1dc18f4 (c1dc18f444603397768fcb7c34a8488b10284df6) reef (stable) 2024-02-21T23:40:37.084 INFO:teuthology.orchestra.run.smithi090.stdout:Extracting ceph user uid/gid from container image... 2024-02-21T23:40:37.668 INFO:teuthology.orchestra.run.smithi090.stdout:stat: stdout 167 167 2024-02-21T23:40:37.668 INFO:teuthology.orchestra.run.smithi090.stdout:Creating initial keys... 2024-02-21T23:40:38.286 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph-authtool: stdout AQD1idZlbkPNNxAAT/HrPKLNJ6Kknsux0QfgEw== 2024-02-21T23:40:38.905 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph-authtool: stdout AQD2idZlDSVtIBAAzB2PCbLqhfFXmPeRDM9QVA== 2024-02-21T23:40:39.503 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph-authtool: stdout AQD3idZlIWcXChAAVqI5MlA+4qoR4GA3qoTFAw== 2024-02-21T23:40:39.504 INFO:teuthology.orchestra.run.smithi090.stdout:Creating initial monmap... 2024-02-21T23:40:40.092 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-21T23:40:40.092 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-02-21T23:40:40.092 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 7ed81218-d112-11ee-95bd-87774f69a715 2024-02-21T23:40:40.092 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-21T23:40:40.092 INFO:teuthology.orchestra.run.smithi090.stdout:monmaptool for a [v2:172.21.15.90:3300,v1:172.21.15.90:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-21T23:40:40.092 INFO:teuthology.orchestra.run.smithi090.stdout:setting min_mon_release = pacific 2024-02-21T23:40:40.092 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/monmaptool: set fsid to 7ed81218-d112-11ee-95bd-87774f69a715 2024-02-21T23:40:40.093 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-21T23:40:40.093 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:40:40.093 INFO:teuthology.orchestra.run.smithi090.stdout:Creating mon... 2024-02-21T23:40:40.732 INFO:teuthology.orchestra.run.smithi090.stdout:create mon.a on 2024-02-21T23:40:40.870 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-02-21T23:40:40.961 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-21T23:40:41.086 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-7ed81218-d112-11ee-95bd-87774f69a715.target → /etc/systemd/system/ceph-7ed81218-d112-11ee-95bd-87774f69a715.target. 2024-02-21T23:40:41.087 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-7ed81218-d112-11ee-95bd-87774f69a715.target → /etc/systemd/system/ceph-7ed81218-d112-11ee-95bd-87774f69a715.target. 2024-02-21T23:40:41.280 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-7ed81218-d112-11ee-95bd-87774f69a715@mon.a 2024-02-21T23:40:41.280 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Failed to reset failed state of unit ceph-7ed81218-d112-11ee-95bd-87774f69a715@mon.a.service: Unit ceph-7ed81218-d112-11ee-95bd-87774f69a715@mon.a.service not loaded. 2024-02-21T23:40:41.371 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-7ed81218-d112-11ee-95bd-87774f69a715.target.wants/ceph-7ed81218-d112-11ee-95bd-87774f69a715@mon.a.service → /etc/systemd/system/ceph-7ed81218-d112-11ee-95bd-87774f69a715@.service. 2024-02-21T23:40:41.748 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-21T23:40:41.748 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout disabled 2024-02-21T23:40:41.756 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-21T23:40:41.756 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout inactive 2024-02-21T23:40:41.756 INFO:teuthology.orchestra.run.smithi090.stdout:firewalld.service is not enabled 2024-02-21T23:40:41.756 INFO:teuthology.orchestra.run.smithi090.stdout:Not possible to enable service . firewalld.service is not available 2024-02-21T23:40:41.757 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for mon to start... 2024-02-21T23:40:41.757 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for mon... 2024-02-21T23:40:41.991 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:41 smithi090 ceph-mon[42900]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-21T23:40:42.625 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout cluster: 2024-02-21T23:40:42.625 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout id: 7ed81218-d112-11ee-95bd-87774f69a715 2024-02-21T23:40:42.625 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-02-21T23:40:42.626 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-02-21T23:40:42.626 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout services: 2024-02-21T23:40:42.626 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.50139s) 2024-02-21T23:40:42.626 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-02-21T23:40:42.626 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-02-21T23:40:42.626 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-02-21T23:40:42.626 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout data: 2024-02-21T23:40:42.626 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-02-21T23:40:42.626 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-02-21T23:40:42.626 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-02-21T23:40:42.626 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout pgs: 2024-02-21T23:40:42.626 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-02-21T23:40:42.626 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.209+0000 7f7dac15a700 1 Processor -- start 2024-02-21T23:40:42.626 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.209+0000 7f7dac15a700 1 -- start start 2024-02-21T23:40:42.626 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.209+0000 7f7dac15a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7da4106490 0x7f7da41068b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:40:42.627 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.210+0000 7f7dac15a700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da4106e80 con 0x7f7da4106490 2024-02-21T23:40:42.627 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.210+0000 7f7da9ef6700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7da4106490 0x7f7da41068b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:40:42.627 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.210+0000 7f7da9ef6700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7da4106490 0x7f7da41068b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35010/0 (socket says 172.21.15.90:35010) 2024-02-21T23:40:42.627 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.210+0000 7f7da9ef6700 1 -- 172.21.15.90:0/2535448847 learned_addr learned my addr 172.21.15.90:0/2535448847 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:40:42.627 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.210+0000 7f7da9ef6700 1 -- 172.21.15.90:0/2535448847 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7da4107690 con 0x7f7da4106490 2024-02-21T23:40:42.627 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.210+0000 7f7da9ef6700 1 --2- 172.21.15.90:0/2535448847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7da4106490 0x7f7da41068b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7d980098a0 tx=0x7f7d98009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85c00573ec4e4445 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:40:42.627 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.211+0000 7f7da8ef4700 1 -- 172.21.15.90:0/2535448847 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7d98004160 con 0x7f7da4106490 2024-02-21T23:40:42.627 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.211+0000 7f7da8ef4700 1 -- 172.21.15.90:0/2535448847 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d980042c0 con 0x7f7da4106490 2024-02-21T23:40:42.627 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.211+0000 7f7da8ef4700 1 -- 172.21.15.90:0/2535448847 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7d980045a0 con 0x7f7da4106490 2024-02-21T23:40:42.627 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.211+0000 7f7dac15a700 1 -- 172.21.15.90:0/2535448847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7da4106490 msgr2=0x7f7da41068b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:40:42.627 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.211+0000 7f7dac15a700 1 --2- 172.21.15.90:0/2535448847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7da4106490 0x7f7da41068b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7d980098a0 tx=0x7f7d98009bb0 comp rx=0 tx=0).stop 2024-02-21T23:40:42.627 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.211+0000 7f7dac15a700 1 -- 172.21.15.90:0/2535448847 shutdown_connections 2024-02-21T23:40:42.627 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.211+0000 7f7dac15a700 1 --2- 172.21.15.90:0/2535448847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7da4106490 0x7f7da41068b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:40:42.628 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.211+0000 7f7dac15a700 1 -- 172.21.15.90:0/2535448847 >> 172.21.15.90:0/2535448847 conn(0x7f7da4101a90 msgr2=0x7f7da4103eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:40:42.628 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.211+0000 7f7dac15a700 1 -- 172.21.15.90:0/2535448847 shutdown_connections 2024-02-21T23:40:42.628 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.211+0000 7f7dac15a700 1 -- 172.21.15.90:0/2535448847 wait complete. 2024-02-21T23:40:42.628 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.212+0000 7f7dac15a700 1 Processor -- start 2024-02-21T23:40:42.628 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.212+0000 7f7dac15a700 1 -- start start 2024-02-21T23:40:42.628 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.212+0000 7f7dac15a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7da4106490 0x7f7da4199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:40:42.628 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.212+0000 7f7dac15a700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da4199e80 con 0x7f7da4106490 2024-02-21T23:40:42.628 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.212+0000 7f7da9ef6700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7da4106490 0x7f7da4199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:40:42.628 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.212+0000 7f7da9ef6700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7da4106490 0x7f7da4199940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35024/0 (socket says 172.21.15.90:35024) 2024-02-21T23:40:42.628 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.212+0000 7f7da9ef6700 1 -- 172.21.15.90:0/1363226833 learned_addr learned my addr 172.21.15.90:0/1363226833 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:40:42.628 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.213+0000 7f7da9ef6700 1 -- 172.21.15.90:0/1363226833 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d98009550 con 0x7f7da4106490 2024-02-21T23:40:42.628 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.213+0000 7f7da9ef6700 1 --2- 172.21.15.90:0/1363226833 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7da4106490 0x7f7da4199940 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7d98004000 tx=0x7f7d98004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:40:42.628 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.213+0000 7f7d96ffd700 1 -- 172.21.15.90:0/1363226833 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7d98011630 con 0x7f7da4106490 2024-02-21T23:40:42.628 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.213+0000 7f7d96ffd700 1 -- 172.21.15.90:0/1363226833 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d98011790 con 0x7f7da4106490 2024-02-21T23:40:42.629 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.213+0000 7f7dac15a700 1 -- 172.21.15.90:0/1363226833 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7da419a080 con 0x7f7da4106490 2024-02-21T23:40:42.629 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.213+0000 7f7d96ffd700 1 -- 172.21.15.90:0/1363226833 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7d98011a30 con 0x7f7da4106490 2024-02-21T23:40:42.629 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.213+0000 7f7dac15a700 1 -- 172.21.15.90:0/1363226833 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7da419a520 con 0x7f7da4106490 2024-02-21T23:40:42.629 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.213+0000 7f7d96ffd700 1 -- 172.21.15.90:0/1363226833 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7d98022070 con 0x7f7da4106490 2024-02-21T23:40:42.629 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.213+0000 7f7d96ffd700 1 -- 172.21.15.90:0/1363226833 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7d9801c450 con 0x7f7da4106490 2024-02-21T23:40:42.629 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.214+0000 7f7dac15a700 1 -- 172.21.15.90:0/1363226833 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7da41936b0 con 0x7f7da4106490 2024-02-21T23:40:42.629 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.216+0000 7f7d96ffd700 1 -- 172.21.15.90:0/1363226833 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f7d9801b450 con 0x7f7da4106490 2024-02-21T23:40:42.629 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.284+0000 7f7dac15a700 1 -- 172.21.15.90:0/1363226833 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f7da402ce90 con 0x7f7da4106490 2024-02-21T23:40:42.629 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.284+0000 7f7d96ffd700 1 -- 172.21.15.90:0/1363226833 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+316 (secure 0 0 0) 0x7f7d9801b5f0 con 0x7f7da4106490 2024-02-21T23:40:42.629 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.285+0000 7f7dac15a700 1 -- 172.21.15.90:0/1363226833 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7da4106490 msgr2=0x7f7da4199940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:40:42.629 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.285+0000 7f7dac15a700 1 --2- 172.21.15.90:0/1363226833 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7da4106490 0x7f7da4199940 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7d98004000 tx=0x7f7d98004750 comp rx=0 tx=0).stop 2024-02-21T23:40:42.629 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.285+0000 7f7dac15a700 1 -- 172.21.15.90:0/1363226833 shutdown_connections 2024-02-21T23:40:42.629 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.285+0000 7f7dac15a700 1 --2- 172.21.15.90:0/1363226833 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7da4106490 0x7f7da4199940 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:40:42.629 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.285+0000 7f7dac15a700 1 -- 172.21.15.90:0/1363226833 >> 172.21.15.90:0/1363226833 conn(0x7f7da4101a90 msgr2=0x7f7da41907f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:40:42.630 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.285+0000 7f7dac15a700 1 -- 172.21.15.90:0/1363226833 shutdown_connections 2024-02-21T23:40:42.630 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:42.285+0000 7f7dac15a700 1 -- 172.21.15.90:0/1363226833 wait complete. 2024-02-21T23:40:42.630 INFO:teuthology.orchestra.run.smithi090.stdout:mon is available 2024-02-21T23:40:42.630 INFO:teuthology.orchestra.run.smithi090.stdout:Assimilating anything we can from ceph.conf... 2024-02-21T23:40:42.800 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:42 smithi090 ceph-mon[42900]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-21T23:40:42.800 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:42 smithi090 ceph-mon[42900]: monmap e1: 1 mons at {a=[v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-21T23:40:42.800 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:42 smithi090 ceph-mon[42900]: fsmap 2024-02-21T23:40:42.800 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:42 smithi090 ceph-mon[42900]: osdmap e1: 0 total, 0 up, 0 in 2024-02-21T23:40:42.800 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:42 smithi090 ceph-mon[42900]: mgrmap e1: no daemons active 2024-02-21T23:40:42.801 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:42 smithi090 ceph-mon[42900]: from='client.? 172.21.15.90:0/1363226833' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-21T23:40:43.486 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-02-21T23:40:43.494 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout [global] 2024-02-21T23:40:43.494 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout fsid = 7ed81218-d112-11ee-95bd-87774f69a715 2024-02-21T23:40:43.494 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.90:3300,v1:172.21.15.90:6789] 2024-02-21T23:40:43.494 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-21T23:40:43.495 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-21T23:40:43.495 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-21T23:40:43.495 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-21T23:40:43.495 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-02-21T23:40:43.495 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-21T23:40:43.495 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-21T23:40:43.495 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-02-21T23:40:43.495 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout [osd] 2024-02-21T23:40:43.495 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-02-21T23:40:43.495 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-02-21T23:40:43.495 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-02-21T23:40:43.495 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-21T23:40:43.496 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-21T23:40:43.496 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.064+0000 7f5f2af46700 1 Processor -- start 2024-02-21T23:40:43.496 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.064+0000 7f5f2af46700 1 -- start start 2024-02-21T23:40:43.496 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.064+0000 7f5f2af46700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f24106f60 0x7f5f24109350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:40:43.496 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.064+0000 7f5f2af46700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f24079f60 con 0x7f5f24106f60 2024-02-21T23:40:43.496 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.064+0000 7f5f28ce2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f24106f60 0x7f5f24109350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:40:43.496 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.064+0000 7f5f28ce2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f24106f60 0x7f5f24109350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35026/0 (socket says 172.21.15.90:35026) 2024-02-21T23:40:43.496 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.064+0000 7f5f28ce2700 1 -- 172.21.15.90:0/2331600553 learned_addr learned my addr 172.21.15.90:0/2331600553 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:40:43.496 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.064+0000 7f5f28ce2700 1 -- 172.21.15.90:0/2331600553 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f24109890 con 0x7f5f24106f60 2024-02-21T23:40:43.496 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.065+0000 7f5f28ce2700 1 --2- 172.21.15.90:0/2331600553 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f24106f60 0x7f5f24109350 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f5f180098a0 tx=0x7f5f18009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37fe7ce06977b721 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:40:43.496 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.065+0000 7f5f237fe700 1 -- 172.21.15.90:0/2331600553 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5f18004160 con 0x7f5f24106f60 2024-02-21T23:40:43.496 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.065+0000 7f5f237fe700 1 -- 172.21.15.90:0/2331600553 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f180042c0 con 0x7f5f24106f60 2024-02-21T23:40:43.496 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.065+0000 7f5f237fe700 1 -- 172.21.15.90:0/2331600553 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5f180045a0 con 0x7f5f24106f60 2024-02-21T23:40:43.497 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.065+0000 7f5f2af46700 1 -- 172.21.15.90:0/2331600553 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f24106f60 msgr2=0x7f5f24109350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:40:43.497 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.065+0000 7f5f2af46700 1 --2- 172.21.15.90:0/2331600553 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f24106f60 0x7f5f24109350 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f5f180098a0 tx=0x7f5f18009bb0 comp rx=0 tx=0).stop 2024-02-21T23:40:43.497 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.065+0000 7f5f2af46700 1 -- 172.21.15.90:0/2331600553 shutdown_connections 2024-02-21T23:40:43.497 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.065+0000 7f5f2af46700 1 --2- 172.21.15.90:0/2331600553 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f24106f60 0x7f5f24109350 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:40:43.497 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.065+0000 7f5f2af46700 1 -- 172.21.15.90:0/2331600553 >> 172.21.15.90:0/2331600553 conn(0x7f5f24100970 msgr2=0x7f5f24102d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:40:43.497 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.065+0000 7f5f2af46700 1 -- 172.21.15.90:0/2331600553 shutdown_connections 2024-02-21T23:40:43.497 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.065+0000 7f5f2af46700 1 -- 172.21.15.90:0/2331600553 wait complete. 2024-02-21T23:40:43.497 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.066+0000 7f5f2af46700 1 Processor -- start 2024-02-21T23:40:43.497 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.066+0000 7f5f2af46700 1 -- start start 2024-02-21T23:40:43.497 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.066+0000 7f5f2af46700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f24106f60 0x7f5f24199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:40:43.497 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.066+0000 7f5f2af46700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f24199fb0 con 0x7f5f24106f60 2024-02-21T23:40:43.497 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.066+0000 7f5f28ce2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f24106f60 0x7f5f24199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:40:43.497 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.066+0000 7f5f28ce2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f24106f60 0x7f5f24199a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35034/0 (socket says 172.21.15.90:35034) 2024-02-21T23:40:43.498 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.066+0000 7f5f28ce2700 1 -- 172.21.15.90:0/1697468565 learned_addr learned my addr 172.21.15.90:0/1697468565 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:40:43.498 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.067+0000 7f5f28ce2700 1 -- 172.21.15.90:0/1697468565 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f18009550 con 0x7f5f24106f60 2024-02-21T23:40:43.498 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.067+0000 7f5f28ce2700 1 --2- 172.21.15.90:0/1697468565 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f24106f60 0x7f5f24199a70 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f5f240792a0 tx=0x7f5f18004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:40:43.498 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.067+0000 7f5f21ffb700 1 -- 172.21.15.90:0/1697468565 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5f18011630 con 0x7f5f24106f60 2024-02-21T23:40:43.498 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.067+0000 7f5f21ffb700 1 -- 172.21.15.90:0/1697468565 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f18011790 con 0x7f5f24106f60 2024-02-21T23:40:43.498 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.067+0000 7f5f2af46700 1 -- 172.21.15.90:0/1697468565 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f2419a1b0 con 0x7f5f24106f60 2024-02-21T23:40:43.498 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.067+0000 7f5f21ffb700 1 -- 172.21.15.90:0/1697468565 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5f18011a30 con 0x7f5f24106f60 2024-02-21T23:40:43.498 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.067+0000 7f5f2af46700 1 -- 172.21.15.90:0/1697468565 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f2419a650 con 0x7f5f24106f60 2024-02-21T23:40:43.498 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.067+0000 7f5f21ffb700 1 -- 172.21.15.90:0/1697468565 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f5f18022070 con 0x7f5f24106f60 2024-02-21T23:40:43.498 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.067+0000 7f5f21ffb700 1 -- 172.21.15.90:0/1697468565 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f5f1801c450 con 0x7f5f24106f60 2024-02-21T23:40:43.498 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.068+0000 7f5f2af46700 1 -- 172.21.15.90:0/1697468565 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f241938b0 con 0x7f5f24106f60 2024-02-21T23:40:43.498 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.070+0000 7f5f21ffb700 1 -- 172.21.15.90:0/1697468565 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f5f1801b450 con 0x7f5f24106f60 2024-02-21T23:40:43.498 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.136+0000 7f5f2af46700 1 -- 172.21.15.90:0/1697468565 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f5f240623a0 con 0x7f5f24106f60 2024-02-21T23:40:43.499 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.140+0000 7f5f21ffb700 1 -- 172.21.15.90:0/1697468565 <== mon.0 v2:172.21.15.90:3300/0 7 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5f1801b5f0 con 0x7f5f24106f60 2024-02-21T23:40:43.499 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.140+0000 7f5f21ffb700 1 -- 172.21.15.90:0/1697468565 <== mon.0 v2:172.21.15.90:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f5f18024630 con 0x7f5f24106f60 2024-02-21T23:40:43.499 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.141+0000 7f5f2af46700 1 -- 172.21.15.90:0/1697468565 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f24106f60 msgr2=0x7f5f24199a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:40:43.499 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.141+0000 7f5f2af46700 1 --2- 172.21.15.90:0/1697468565 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f24106f60 0x7f5f24199a70 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f5f240792a0 tx=0x7f5f18004750 comp rx=0 tx=0).stop 2024-02-21T23:40:43.499 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.141+0000 7f5f2af46700 1 -- 172.21.15.90:0/1697468565 shutdown_connections 2024-02-21T23:40:43.499 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.141+0000 7f5f2af46700 1 --2- 172.21.15.90:0/1697468565 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f24106f60 0x7f5f24199a70 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:40:43.499 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.141+0000 7f5f2af46700 1 -- 172.21.15.90:0/1697468565 >> 172.21.15.90:0/1697468565 conn(0x7f5f24100970 msgr2=0x7f5f24102cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:40:43.499 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.141+0000 7f5f2af46700 1 -- 172.21.15.90:0/1697468565 shutdown_connections 2024-02-21T23:40:43.499 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.141+0000 7f5f2af46700 1 -- 172.21.15.90:0/1697468565 wait complete. 2024-02-21T23:40:43.499 INFO:teuthology.orchestra.run.smithi090.stdout:Generating new minimal ceph.conf... 2024-02-21T23:40:44.068 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:43 smithi090 ceph-mon[42900]: from='client.? 172.21.15.90:0/1697468565' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-21T23:40:44.068 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:43 smithi090 ceph-mon[42900]: from='client.? 172.21.15.90:0/1697468565' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-21T23:40:44.257 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.840+0000 7f34f72b5700 1 Processor -- start 2024-02-21T23:40:44.257 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.840+0000 7f34f72b5700 1 -- start start 2024-02-21T23:40:44.257 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.840+0000 7f34f72b5700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34f0106f00 0x7f34f01092f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:40:44.257 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.840+0000 7f34f72b5700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f0109830 con 0x7f34f0106f00 2024-02-21T23:40:44.257 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.840+0000 7f34f5051700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34f0106f00 0x7f34f01092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:40:44.257 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.840+0000 7f34f5051700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34f0106f00 0x7f34f01092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35036/0 (socket says 172.21.15.90:35036) 2024-02-21T23:40:44.257 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.840+0000 7f34f5051700 1 -- 172.21.15.90:0/3513566110 learned_addr learned my addr 172.21.15.90:0/3513566110 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:40:44.257 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.841+0000 7f34f5051700 1 -- 172.21.15.90:0/3513566110 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f0109970 con 0x7f34f0106f00 2024-02-21T23:40:44.258 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.841+0000 7f34f5051700 1 --2- 172.21.15.90:0/3513566110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34f0106f00 0x7f34f01092f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f34ec0098a0 tx=0x7f34ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adc5c5136f06b193 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:40:44.258 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.841+0000 7f34e7fff700 1 -- 172.21.15.90:0/3513566110 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f34ec004160 con 0x7f34f0106f00 2024-02-21T23:40:44.258 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.841+0000 7f34e7fff700 1 -- 172.21.15.90:0/3513566110 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f34ec0042c0 con 0x7f34f0106f00 2024-02-21T23:40:44.258 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.841+0000 7f34e7fff700 1 -- 172.21.15.90:0/3513566110 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f34ec011a00 con 0x7f34f0106f00 2024-02-21T23:40:44.258 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.841+0000 7f34f72b5700 1 -- 172.21.15.90:0/3513566110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34f0106f00 msgr2=0x7f34f01092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:40:44.258 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.841+0000 7f34f72b5700 1 --2- 172.21.15.90:0/3513566110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34f0106f00 0x7f34f01092f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f34ec0098a0 tx=0x7f34ec009bb0 comp rx=0 tx=0).stop 2024-02-21T23:40:44.258 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.841+0000 7f34f72b5700 1 -- 172.21.15.90:0/3513566110 shutdown_connections 2024-02-21T23:40:44.258 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.841+0000 7f34f72b5700 1 --2- 172.21.15.90:0/3513566110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34f0106f00 0x7f34f01092f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:40:44.258 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.841+0000 7f34f72b5700 1 -- 172.21.15.90:0/3513566110 >> 172.21.15.90:0/3513566110 conn(0x7f34f0100970 msgr2=0x7f34f0102d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:40:44.258 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.842+0000 7f34f72b5700 1 -- 172.21.15.90:0/3513566110 shutdown_connections 2024-02-21T23:40:44.258 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.842+0000 7f34f72b5700 1 -- 172.21.15.90:0/3513566110 wait complete. 2024-02-21T23:40:44.258 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.842+0000 7f34f72b5700 1 Processor -- start 2024-02-21T23:40:44.258 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.842+0000 7f34f72b5700 1 -- start start 2024-02-21T23:40:44.258 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.842+0000 7f34f72b5700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34f0106f00 0x7f34f0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:40:44.259 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.842+0000 7f34f72b5700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f0106c70 con 0x7f34f0106f00 2024-02-21T23:40:44.259 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.842+0000 7f34f5051700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34f0106f00 0x7f34f0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:40:44.259 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.843+0000 7f34f5051700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34f0106f00 0x7f34f0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35038/0 (socket says 172.21.15.90:35038) 2024-02-21T23:40:44.259 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.843+0000 7f34f5051700 1 -- 172.21.15.90:0/3917412614 learned_addr learned my addr 172.21.15.90:0/3917412614 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:40:44.259 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.843+0000 7f34f5051700 1 -- 172.21.15.90:0/3917412614 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34ec009550 con 0x7f34f0106f00 2024-02-21T23:40:44.259 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.843+0000 7f34f5051700 1 --2- 172.21.15.90:0/3917412614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34f0106f00 0x7f34f0106730 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f34ec009130 tx=0x7f34ec011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:40:44.259 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.843+0000 7f34e67fc700 1 -- 172.21.15.90:0/3917412614 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f34ec0038b0 con 0x7f34f0106f00 2024-02-21T23:40:44.259 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.843+0000 7f34e67fc700 1 -- 172.21.15.90:0/3917412614 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f34ec003a10 con 0x7f34f0106f00 2024-02-21T23:40:44.259 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.843+0000 7f34f72b5700 1 -- 172.21.15.90:0/3917412614 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34f0104eb0 con 0x7f34f0106f00 2024-02-21T23:40:44.259 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.843+0000 7f34e67fc700 1 -- 172.21.15.90:0/3917412614 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f34ec019550 con 0x7f34f0106f00 2024-02-21T23:40:44.259 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.843+0000 7f34f72b5700 1 -- 172.21.15.90:0/3917412614 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34f0105350 con 0x7f34f0106f00 2024-02-21T23:40:44.259 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.843+0000 7f34e67fc700 1 -- 172.21.15.90:0/3917412614 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f34ec019ae0 con 0x7f34f0106f00 2024-02-21T23:40:44.260 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.843+0000 7f34e67fc700 1 -- 172.21.15.90:0/3917412614 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f34ec02dde0 con 0x7f34f0106f00 2024-02-21T23:40:44.260 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.844+0000 7f34f72b5700 1 -- 172.21.15.90:0/3917412614 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34f00623a0 con 0x7f34f0106f00 2024-02-21T23:40:44.260 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.846+0000 7f34e67fc700 1 -- 172.21.15.90:0/3917412614 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f34ec003b80 con 0x7f34f0106f00 2024-02-21T23:40:44.260 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.913+0000 7f34f72b5700 1 -- 172.21.15.90:0/3917412614 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f34f0105d30 con 0x7f34f0106f00 2024-02-21T23:40:44.260 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.913+0000 7f34e67fc700 1 -- 172.21.15.90:0/3917412614 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f34ec01a960 con 0x7f34f0106f00 2024-02-21T23:40:44.260 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.914+0000 7f34f72b5700 1 -- 172.21.15.90:0/3917412614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34f0106f00 msgr2=0x7f34f0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:40:44.260 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.914+0000 7f34f72b5700 1 --2- 172.21.15.90:0/3917412614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34f0106f00 0x7f34f0106730 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f34ec009130 tx=0x7f34ec011ad0 comp rx=0 tx=0).stop 2024-02-21T23:40:44.260 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.914+0000 7f34f72b5700 1 -- 172.21.15.90:0/3917412614 shutdown_connections 2024-02-21T23:40:44.260 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.914+0000 7f34f72b5700 1 --2- 172.21.15.90:0/3917412614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34f0106f00 0x7f34f0106730 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:40:44.261 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.914+0000 7f34f72b5700 1 -- 172.21.15.90:0/3917412614 >> 172.21.15.90:0/3917412614 conn(0x7f34f0100970 msgr2=0x7f34f0101540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:40:44.261 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.914+0000 7f34f72b5700 1 -- 172.21.15.90:0/3917412614 shutdown_connections 2024-02-21T23:40:44.261 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:43.914+0000 7f34f72b5700 1 -- 172.21.15.90:0/3917412614 wait complete. 2024-02-21T23:40:44.261 INFO:teuthology.orchestra.run.smithi090.stdout:Restarting the monitor... 2024-02-21T23:40:44.456 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:44 smithi090 systemd[1]: Stopping Ceph mon.a for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:40:44.457 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:44 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mon-a[42877]: 2024-02-21T23:40:44.470+0000 7fb074031700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-21T23:40:44.710 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:44 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mon-a[42877]: 2024-02-21T23:40:44.470+0000 7fb074031700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-02-21T23:40:44.710 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:44 smithi090 bash[43226]: ceph-7ed81218-d112-11ee-95bd-87774f69a715-mon-a 2024-02-21T23:40:45.109 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:45 smithi090 systemd[1]: ceph-7ed81218-d112-11ee-95bd-87774f69a715@mon.a.service: Succeeded. 2024-02-21T23:40:45.109 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:45 smithi090 systemd[1]: Stopped Ceph mon.a for 7ed81218-d112-11ee-95bd-87774f69a715. 2024-02-21T23:40:45.109 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:45 smithi090 systemd[1]: Starting Ceph mon.a for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:40:46.781 INFO:teuthology.orchestra.run.smithi090.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-02-21T23:40:46.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: set uid:gid to 167:167 (ceph:ceph) 2024-02-21T23:40:46.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: ceph version 18.2.1-638-gc1dc18f4 (c1dc18f444603397768fcb7c34a8488b10284df6) reef (stable), process ceph-mon, pid 7 2024-02-21T23:40:46.981 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: pidfile_write: ignore empty --pid-file 2024-02-21T23:40:46.981 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: load: jerasure load: lrc 2024-02-21T23:40:46.981 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: RocksDB version: 7.9.2 2024-02-21T23:40:46.981 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Git sha 0 2024-02-21T23:40:46.981 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Compile date 2024-02-21 21:10:13 2024-02-21T23:40:46.981 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: DB SUMMARY 2024-02-21T23:40:46.981 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: DB Session ID: T9HP3OUQHLS03VEW6USX 2024-02-21T23:40:46.982 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: CURRENT file: CURRENT 2024-02-21T23:40:46.982 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: IDENTITY file: IDENTITY 2024-02-21T23:40:46.982 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-02-21T23:40:46.982 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-02-21T23:40:46.982 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 97418 ; 2024-02-21T23:40:46.982 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.error_if_exists: 0 2024-02-21T23:40:46.983 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.create_if_missing: 0 2024-02-21T23:40:46.983 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.paranoid_checks: 1 2024-02-21T23:40:46.983 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-21T23:40:46.983 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-21T23:40:46.983 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-21T23:40:46.983 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.env: 0x5640fff5e720 2024-02-21T23:40:46.984 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.fs: PosixFileSystem 2024-02-21T23:40:46.984 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.info_log: 0x5641032b94c0 2024-02-21T23:40:46.984 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.max_file_opening_threads: 16 2024-02-21T23:40:46.984 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.statistics: (nil) 2024-02-21T23:40:46.984 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.use_fsync: 0 2024-02-21T23:40:46.984 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.max_log_file_size: 0 2024-02-21T23:40:46.985 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-21T23:40:46.985 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-21T23:40:46.985 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.keep_log_file_num: 1000 2024-02-21T23:40:46.985 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.recycle_log_file_num: 0 2024-02-21T23:40:46.985 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.allow_fallocate: 1 2024-02-21T23:40:46.985 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.allow_mmap_reads: 0 2024-02-21T23:40:46.985 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.allow_mmap_writes: 0 2024-02-21T23:40:46.985 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.use_direct_reads: 0 2024-02-21T23:40:46.986 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-21T23:40:46.986 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.create_missing_column_families: 0 2024-02-21T23:40:46.986 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.db_log_dir: 2024-02-21T23:40:46.986 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.wal_dir: 2024-02-21T23:40:46.986 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-21T23:40:46.986 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-21T23:40:46.986 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-21T23:40:46.987 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-21T23:40:46.987 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-21T23:40:46.987 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-21T23:40:46.987 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.advise_random_on_open: 1 2024-02-21T23:40:46.987 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.db_write_buffer_size: 0 2024-02-21T23:40:46.987 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.write_buffer_manager: 0x56410254a320 2024-02-21T23:40:46.988 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-21T23:40:46.988 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-21T23:40:46.988 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-21T23:40:46.988 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.rate_limiter: (nil) 2024-02-21T23:40:46.988 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-21T23:40:46.989 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.wal_recovery_mode: 2 2024-02-21T23:40:46.989 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.enable_thread_tracking: 0 2024-02-21T23:40:46.989 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.enable_pipelined_write: 0 2024-02-21T23:40:46.989 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.unordered_write: 0 2024-02-21T23:40:46.989 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-21T23:40:46.989 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-21T23:40:46.990 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-21T23:40:46.990 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-21T23:40:46.990 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.row_cache: None 2024-02-21T23:40:46.990 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.wal_filter: None 2024-02-21T23:40:46.990 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-21T23:40:46.990 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.allow_ingest_behind: 0 2024-02-21T23:40:46.990 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.two_write_queues: 0 2024-02-21T23:40:46.991 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.manual_wal_flush: 0 2024-02-21T23:40:46.991 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.wal_compression: 0 2024-02-21T23:40:46.991 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.atomic_flush: 0 2024-02-21T23:40:46.991 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-21T23:40:46.991 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-21T23:40:46.991 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-21T23:40:46.991 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.log_readahead_size: 0 2024-02-21T23:40:46.992 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-21T23:40:46.992 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.best_efforts_recovery: 0 2024-02-21T23:40:46.992 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-21T23:40:46.992 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-21T23:40:46.992 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.allow_data_in_errors: 0 2024-02-21T23:40:46.992 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.db_host_id: __hostname__ 2024-02-21T23:40:46.993 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-21T23:40:46.993 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.max_background_jobs: 2 2024-02-21T23:40:46.993 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.max_background_compactions: -1 2024-02-21T23:40:46.993 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.max_subcompactions: 1 2024-02-21T23:40:46.993 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-21T23:40:46.993 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-21T23:40:46.993 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-21T23:40:46.994 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.max_total_wal_size: 0 2024-02-21T23:40:46.994 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-21T23:40:46.994 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-21T23:40:46.995 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-21T23:40:46.995 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-21T23:40:46.995 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.max_open_files: -1 2024-02-21T23:40:46.995 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.bytes_per_sync: 0 2024-02-21T23:40:46.995 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-21T23:40:46.995 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-21T23:40:46.995 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.compaction_readahead_size: 0 2024-02-21T23:40:46.996 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.max_background_flushes: -1 2024-02-21T23:40:46.996 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Compression algorithms supported: 2024-02-21T23:40:46.996 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-21T23:40:46.996 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: kZSTD supported: 0 2024-02-21T23:40:46.996 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: kXpressCompression supported: 0 2024-02-21T23:40:46.996 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: kLZ4HCCompression supported: 1 2024-02-21T23:40:46.996 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: kZlibCompression supported: 1 2024-02-21T23:40:46.996 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: kSnappyCompression supported: 1 2024-02-21T23:40:46.996 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: kLZ4Compression supported: 1 2024-02-21T23:40:46.996 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: kBZip2Compression supported: 0 2024-02-21T23:40:46.996 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-21T23:40:46.996 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-21T23:40:46.997 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-02-21T23:40:46.997 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-21T23:40:46.997 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-21T23:40:46.997 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.merge_operator: 2024-02-21T23:40:46.997 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.compaction_filter: None 2024-02-21T23:40:46.997 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.compaction_filter_factory: None 2024-02-21T23:40:46.997 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.sst_partitioner_factory: None 2024-02-21T23:40:46.997 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-21T23:40:46.997 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-21T23:40:46.997 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5641032b95c0) 2024-02-21T23:40:46.997 INFO:journalctl@ceph.mon.a.smithi090.stdout: cache_index_and_filter_blocks: 1 2024-02-21T23:40:46.997 INFO:journalctl@ceph.mon.a.smithi090.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-21T23:40:46.997 INFO:journalctl@ceph.mon.a.smithi090.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-21T23:40:46.998 INFO:journalctl@ceph.mon.a.smithi090.stdout: pin_top_level_index_and_filter: 1 2024-02-21T23:40:46.998 INFO:journalctl@ceph.mon.a.smithi090.stdout: index_type: 0 2024-02-21T23:40:46.998 INFO:journalctl@ceph.mon.a.smithi090.stdout: data_block_index_type: 0 2024-02-21T23:40:46.998 INFO:journalctl@ceph.mon.a.smithi090.stdout: index_shortening: 1 2024-02-21T23:40:46.998 INFO:journalctl@ceph.mon.a.smithi090.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-21T23:40:46.998 INFO:journalctl@ceph.mon.a.smithi090.stdout: checksum: 4 2024-02-21T23:40:46.998 INFO:journalctl@ceph.mon.a.smithi090.stdout: no_block_cache: 0 2024-02-21T23:40:46.998 INFO:journalctl@ceph.mon.a.smithi090.stdout: block_cache: 0x5641025cb1f0 2024-02-21T23:40:46.998 INFO:journalctl@ceph.mon.a.smithi090.stdout: block_cache_name: BinnedLRUCache 2024-02-21T23:40:46.998 INFO:journalctl@ceph.mon.a.smithi090.stdout: block_cache_options: 2024-02-21T23:40:46.998 INFO:journalctl@ceph.mon.a.smithi090.stdout: capacity : 536870912 2024-02-21T23:40:46.998 INFO:journalctl@ceph.mon.a.smithi090.stdout: num_shard_bits : 4 2024-02-21T23:40:46.998 INFO:journalctl@ceph.mon.a.smithi090.stdout: strict_capacity_limit : 0 2024-02-21T23:40:46.999 INFO:journalctl@ceph.mon.a.smithi090.stdout: high_pri_pool_ratio: 0.000 2024-02-21T23:40:46.999 INFO:journalctl@ceph.mon.a.smithi090.stdout: block_cache_compressed: (nil) 2024-02-21T23:40:46.999 INFO:journalctl@ceph.mon.a.smithi090.stdout: persistent_cache: (nil) 2024-02-21T23:40:46.999 INFO:journalctl@ceph.mon.a.smithi090.stdout: block_size: 4096 2024-02-21T23:40:46.999 INFO:journalctl@ceph.mon.a.smithi090.stdout: block_size_deviation: 10 2024-02-21T23:40:46.999 INFO:journalctl@ceph.mon.a.smithi090.stdout: block_restart_interval: 16 2024-02-21T23:40:46.999 INFO:journalctl@ceph.mon.a.smithi090.stdout: index_block_restart_interval: 1 2024-02-21T23:40:46.999 INFO:journalctl@ceph.mon.a.smithi090.stdout: metadata_block_size: 4096 2024-02-21T23:40:46.999 INFO:journalctl@ceph.mon.a.smithi090.stdout: partition_filters: 0 2024-02-21T23:40:46.999 INFO:journalctl@ceph.mon.a.smithi090.stdout: use_delta_encoding: 1 2024-02-21T23:40:46.999 INFO:journalctl@ceph.mon.a.smithi090.stdout: filter_policy: bloomfilter 2024-02-21T23:40:46.999 INFO:journalctl@ceph.mon.a.smithi090.stdout: whole_key_filtering: 1 2024-02-21T23:40:46.999 INFO:journalctl@ceph.mon.a.smithi090.stdout: verify_compression: 0 2024-02-21T23:40:46.999 INFO:journalctl@ceph.mon.a.smithi090.stdout: read_amp_bytes_per_bit: 0 2024-02-21T23:40:47.000 INFO:journalctl@ceph.mon.a.smithi090.stdout: format_version: 5 2024-02-21T23:40:47.000 INFO:journalctl@ceph.mon.a.smithi090.stdout: enable_index_compression: 1 2024-02-21T23:40:47.000 INFO:journalctl@ceph.mon.a.smithi090.stdout: block_align: 0 2024-02-21T23:40:47.000 INFO:journalctl@ceph.mon.a.smithi090.stdout: max_auto_readahead_size: 262144 2024-02-21T23:40:47.000 INFO:journalctl@ceph.mon.a.smithi090.stdout: prepopulate_block_cache: 0 2024-02-21T23:40:47.000 INFO:journalctl@ceph.mon.a.smithi090.stdout: initial_auto_readahead_size: 8192 2024-02-21T23:40:47.000 INFO:journalctl@ceph.mon.a.smithi090.stdout: num_file_reads_for_auto_readahead: 2 2024-02-21T23:40:47.000 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.write_buffer_size: 33554432 2024-02-21T23:40:47.000 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.max_write_buffer_number: 2 2024-02-21T23:40:47.000 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.compression: NoCompression 2024-02-21T23:40:47.000 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.bottommost_compression: Disabled 2024-02-21T23:40:47.000 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.prefix_extractor: nullptr 2024-02-21T23:40:47.000 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-21T23:40:47.000 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.num_levels: 7 2024-02-21T23:40:47.001 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-21T23:40:47.001 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-21T23:40:47.001 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-21T23:40:47.001 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-21T23:40:47.001 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-21T23:40:47.001 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-21T23:40:47.001 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-21T23:40:47.001 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-21T23:40:47.001 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-21T23:40:47.001 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-21T23:40:47.001 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-21T23:40:47.001 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-21T23:40:47.001 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-21T23:40:47.001 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.compression_opts.level: 32767 2024-02-21T23:40:47.002 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.compression_opts.strategy: 0 2024-02-21T23:40:47.002 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-21T23:40:47.002 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-21T23:40:47.002 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-21T23:40:47.002 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-21T23:40:47.002 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.compression_opts.enabled: false 2024-02-21T23:40:47.002 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-21T23:40:47.002 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-21T23:40:47.002 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-21T23:40:47.002 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-21T23:40:47.002 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.target_file_size_base: 67108864 2024-02-21T23:40:47.002 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-21T23:40:47.002 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-21T23:40:47.002 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-21T23:40:47.003 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-21T23:40:47.003 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-21T23:40:47.003 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-21T23:40:47.003 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-21T23:40:47.003 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-21T23:40:47.003 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-21T23:40:47.003 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-21T23:40:47.003 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-21T23:40:47.003 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-21T23:40:47.003 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-21T23:40:47.003 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-21T23:40:47.003 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.arena_block_size: 1048576 2024-02-21T23:40:47.003 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-21T23:40:47.003 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-21T23:40:47.003 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.disable_auto_compactions: 0 2024-02-21T23:40:47.004 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-21T23:40:47.004 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-21T23:40:47.004 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-21T23:40:47.004 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-21T23:40:47.004 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-21T23:40:47.004 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-21T23:40:47.004 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-21T23:40:47.004 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-21T23:40:47.004 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-21T23:40:47.004 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-21T23:40:47.004 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.table_properties_collectors: 2024-02-21T23:40:47.004 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.inplace_update_support: 0 2024-02-21T23:40:47.004 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-21T23:40:47.005 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-21T23:40:47.005 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-21T23:40:47.005 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-21T23:40:47.005 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.bloom_locality: 0 2024-02-21T23:40:47.005 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.max_successive_merges: 0 2024-02-21T23:40:47.005 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-21T23:40:47.005 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.paranoid_file_checks: 0 2024-02-21T23:40:47.005 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.force_consistency_checks: 1 2024-02-21T23:40:47.005 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.report_bg_io_stats: 0 2024-02-21T23:40:47.005 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.ttl: 2592000 2024-02-21T23:40:47.005 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-21T23:40:47.005 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-21T23:40:47.005 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-21T23:40:47.005 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.enable_blob_files: false 2024-02-21T23:40:47.005 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.min_blob_size: 0 2024-02-21T23:40:47.006 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.blob_file_size: 268435456 2024-02-21T23:40:47.006 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-21T23:40:47.006 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-21T23:40:47.006 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-21T23:40:47.006 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-21T23:40:47.006 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-21T23:40:47.006 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.blob_file_starting_level: 0 2024-02-21T23:40:47.006 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-21T23:40:47.006 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-02-21T23:40:47.006 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-02-21T23:40:47.006 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 52f61801-d729-45da-b10e-0d3c2bcc9dfd 2024-02-21T23:40:47.006 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708558846682897, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-02-21T23:40:47.006 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-02-21T23:40:47.006 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708558846684070, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 92823, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 307, "table_properties": {"data_size": 90828, "index_size": 277, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13584, "raw_average_key_size": 50, "raw_value_size": 83432, "raw_average_value_size": 309, "num_data_blocks": 11, "num_entries": 270, "num_filter_entries": 270, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1708558846, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "52f61801-d729-45da-b10e-0d3c2bcc9dfd", "db_session_id": "T9HP3OUQHLS03VEW6USX", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-02-21T23:40:47.007 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708558846684151, "job": 1, "event": "recovery_finished"} 2024-02-21T23:40:47.007 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-02-21T23:40:47.007 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-21T23:40:47.007 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x564102668000 2024-02-21T23:40:47.007 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: DB pointer 0x564102656000 2024-02-21T23:40:47.007 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-21T23:40:47.007 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-21T23:40:47.007 INFO:journalctl@ceph.mon.a.smithi090.stdout: ** DB Stats ** 2024-02-21T23:40:47.007 INFO:journalctl@ceph.mon.a.smithi090.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-21T23:40:47.007 INFO:journalctl@ceph.mon.a.smithi090.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-21T23:40:47.007 INFO:journalctl@ceph.mon.a.smithi090.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-21T23:40:47.007 INFO:journalctl@ceph.mon.a.smithi090.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-21T23:40:47.007 INFO:journalctl@ceph.mon.a.smithi090.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-21T23:40:47.007 INFO:journalctl@ceph.mon.a.smithi090.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-21T23:40:47.008 INFO:journalctl@ceph.mon.a.smithi090.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-21T23:40:47.008 INFO:journalctl@ceph.mon.a.smithi090.stdout: 2024-02-21T23:40:47.008 INFO:journalctl@ceph.mon.a.smithi090.stdout: ** Compaction Stats [default] ** 2024-02-21T23:40:47.008 INFO:journalctl@ceph.mon.a.smithi090.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-21T23:40:47.008 INFO:journalctl@ceph.mon.a.smithi090.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-21T23:40:47.008 INFO:journalctl@ceph.mon.a.smithi090.stdout: L0 2/0 92.48 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 119.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-21T23:40:47.008 INFO:journalctl@ceph.mon.a.smithi090.stdout: Sum 2/0 92.48 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 119.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-21T23:40:47.008 INFO:journalctl@ceph.mon.a.smithi090.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 119.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-21T23:40:47.008 INFO:journalctl@ceph.mon.a.smithi090.stdout: 2024-02-21T23:40:47.008 INFO:journalctl@ceph.mon.a.smithi090.stdout: ** Compaction Stats [default] ** 2024-02-21T23:40:47.008 INFO:journalctl@ceph.mon.a.smithi090.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-21T23:40:47.008 INFO:journalctl@ceph.mon.a.smithi090.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-21T23:40:47.008 INFO:journalctl@ceph.mon.a.smithi090.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 119.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-21T23:40:47.009 INFO:journalctl@ceph.mon.a.smithi090.stdout: 2024-02-21T23:40:47.009 INFO:journalctl@ceph.mon.a.smithi090.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-21T23:40:47.009 INFO:journalctl@ceph.mon.a.smithi090.stdout: 2024-02-21T23:40:47.009 INFO:journalctl@ceph.mon.a.smithi090.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-21T23:40:47.009 INFO:journalctl@ceph.mon.a.smithi090.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-21T23:40:47.009 INFO:journalctl@ceph.mon.a.smithi090.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-21T23:40:47.009 INFO:journalctl@ceph.mon.a.smithi090.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-21T23:40:47.009 INFO:journalctl@ceph.mon.a.smithi090.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-21T23:40:47.009 INFO:journalctl@ceph.mon.a.smithi090.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-21T23:40:47.009 INFO:journalctl@ceph.mon.a.smithi090.stdout: Cumulative compaction: 0.00 GB write, 24.78 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-21T23:40:47.009 INFO:journalctl@ceph.mon.a.smithi090.stdout: Interval compaction: 0.00 GB write, 24.78 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-21T23:40:47.009 INFO:journalctl@ceph.mon.a.smithi090.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-21T23:40:47.009 INFO:journalctl@ceph.mon.a.smithi090.stdout: Block cache BinnedLRUCache@0x5641025cb1f0#7 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-02-21T23:40:47.009 INFO:journalctl@ceph.mon.a.smithi090.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-02-21T23:40:47.009 INFO:journalctl@ceph.mon.a.smithi090.stdout: 2024-02-21T23:40:47.010 INFO:journalctl@ceph.mon.a.smithi090.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-21T23:40:47.010 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: starting mon.a rank 0 at public addrs [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] at bind addrs [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 7ed81218-d112-11ee-95bd-87774f69a715 2024-02-21T23:40:47.010 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: mon.a@-1(???) e1 preinit fsid 7ed81218-d112-11ee-95bd-87774f69a715 2024-02-21T23:40:47.010 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: mon.a@-1(???).mds e1 new map 2024-02-21T23:40:47.010 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: mon.a@-1(???).mds e1 print_map 2024-02-21T23:40:47.010 INFO:journalctl@ceph.mon.a.smithi090.stdout: e1 2024-02-21T23:40:47.010 INFO:journalctl@ceph.mon.a.smithi090.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-21T23:40:47.010 INFO:journalctl@ceph.mon.a.smithi090.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-21T23:40:47.010 INFO:journalctl@ceph.mon.a.smithi090.stdout: legacy client fscid: -1 2024-02-21T23:40:47.010 INFO:journalctl@ceph.mon.a.smithi090.stdout: 2024-02-21T23:40:47.010 INFO:journalctl@ceph.mon.a.smithi090.stdout: No filesystems configured 2024-02-21T23:40:47.010 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-21T23:40:47.010 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-21T23:40:47.010 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-21T23:40:47.010 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-21T23:40:47.011 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-21T23:40:47.011 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-21T23:40:47.011 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: monmap e1: 1 mons at {a=[v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-21T23:40:47.011 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: fsmap 2024-02-21T23:40:47.011 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: osdmap e1: 0 total, 0 up, 0 in 2024-02-21T23:40:47.011 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 ceph-mon[43385]: mgrmap e1: no daemons active 2024-02-21T23:40:47.011 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 bash[43347]: d0fe242b867228e90c4bc575eb2a7eb418317970a69971a7eb6d8930295b2005 2024-02-21T23:40:47.011 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:46 smithi090 systemd[1]: Started Ceph mon.a for 7ed81218-d112-11ee-95bd-87774f69a715. 2024-02-21T23:40:49.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:49 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/3244113781' entity='client.admin' 2024-02-21T23:40:51.063 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:47.975+0000 7f3485781700 1 Processor -- start 2024-02-21T23:40:51.068 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:47.975+0000 7f3485781700 1 -- start start 2024-02-21T23:40:51.068 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:47.975+0000 7f3485781700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3480107b90 0x7f3480107fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:40:51.068 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:47.975+0000 7f3485781700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3480108580 con 0x7f3480107b90 2024-02-21T23:40:51.068 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:47.976+0000 7f347effd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3480107b90 0x7f3480107fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:40:51.068 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:47.976+0000 7f347effd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3480107b90 0x7f3480107fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35046/0 (socket says 172.21.15.90:35046) 2024-02-21T23:40:51.068 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:47.976+0000 7f347effd700 1 -- 172.21.15.90:0/613940055 learned_addr learned my addr 172.21.15.90:0/613940055 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:40:51.068 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:47.976+0000 7f347effd700 1 -- 172.21.15.90:0/613940055 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34801060d0 con 0x7f3480107b90 2024-02-21T23:40:51.069 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:47.976+0000 7f347effd700 1 --2- 172.21.15.90:0/613940055 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3480107b90 0x7f3480107fb0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f34740098a0 tx=0x7f3474009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7f5508c75249c1f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:40:51.069 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:47.976+0000 7f347dffb700 1 -- 172.21.15.90:0/613940055 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3474004160 con 0x7f3480107b90 2024-02-21T23:40:51.069 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:47.976+0000 7f347dffb700 1 -- 172.21.15.90:0/613940055 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f34740042c0 con 0x7f3480107b90 2024-02-21T23:40:51.069 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:47.976+0000 7f347dffb700 1 -- 172.21.15.90:0/613940055 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3474011a00 con 0x7f3480107b90 2024-02-21T23:40:51.069 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:47.977+0000 7f3485781700 1 -- 172.21.15.90:0/613940055 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3480107b90 msgr2=0x7f3480107fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:40:51.069 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:47.977+0000 7f3485781700 1 --2- 172.21.15.90:0/613940055 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3480107b90 0x7f3480107fb0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f34740098a0 tx=0x7f3474009bb0 comp rx=0 tx=0).stop 2024-02-21T23:40:51.069 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:47.977+0000 7f3485781700 1 -- 172.21.15.90:0/613940055 shutdown_connections 2024-02-21T23:40:51.069 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:47.977+0000 7f3485781700 1 --2- 172.21.15.90:0/613940055 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3480107b90 0x7f3480107fb0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:40:51.069 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:47.977+0000 7f3485781700 1 -- 172.21.15.90:0/613940055 >> 172.21.15.90:0/613940055 conn(0x7f3480101a70 msgr2=0x7f3480103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:40:51.069 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:47.977+0000 7f3485781700 1 -- 172.21.15.90:0/613940055 shutdown_connections 2024-02-21T23:40:51.069 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:47.977+0000 7f3485781700 1 -- 172.21.15.90:0/613940055 wait complete. 2024-02-21T23:40:51.069 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:47.977+0000 7f3485781700 1 Processor -- start 2024-02-21T23:40:51.069 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:47.977+0000 7f3485781700 1 -- start start 2024-02-21T23:40:51.070 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:47.978+0000 7f3485781700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3480107b90 0x7f348019a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:40:51.070 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:47.978+0000 7f3485781700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348019a840 con 0x7f3480107b90 2024-02-21T23:40:51.070 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:47.978+0000 7f347effd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3480107b90 0x7f348019a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:40:51.070 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:47.978+0000 7f347effd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3480107b90 0x7f348019a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35050/0 (socket says 172.21.15.90:35050) 2024-02-21T23:40:51.070 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:47.978+0000 7f347effd700 1 -- 172.21.15.90:0/3244113781 learned_addr learned my addr 172.21.15.90:0/3244113781 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:40:51.070 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:47.978+0000 7f347effd700 1 -- 172.21.15.90:0/3244113781 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3474009550 con 0x7f3480107b90 2024-02-21T23:40:51.070 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:47.978+0000 7f347effd700 1 --2- 172.21.15.90:0/3244113781 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3480107b90 0x7f348019a300 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f3474011710 tx=0x7f3474011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:40:51.070 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:47.978+0000 7f3467fff700 1 -- 172.21.15.90:0/3244113781 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3474003aa0 con 0x7f3480107b90 2024-02-21T23:40:51.070 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:47.978+0000 7f3485781700 1 -- 172.21.15.90:0/3244113781 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f348019aa40 con 0x7f3480107b90 2024-02-21T23:40:51.070 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:47.978+0000 7f3467fff700 1 -- 172.21.15.90:0/3244113781 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3474003c00 con 0x7f3480107b90 2024-02-21T23:40:51.070 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:47.978+0000 7f3467fff700 1 -- 172.21.15.90:0/3244113781 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3474024800 con 0x7f3480107b90 2024-02-21T23:40:51.070 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:47.978+0000 7f3485781700 1 -- 172.21.15.90:0/3244113781 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f348019aee0 con 0x7f3480107b90 2024-02-21T23:40:51.070 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:47.978+0000 7f3467fff700 1 -- 172.21.15.90:0/3244113781 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f3474024d30 con 0x7f3480107b90 2024-02-21T23:40:51.070 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:47.979+0000 7f3467fff700 1 -- 172.21.15.90:0/3244113781 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f34740193c0 con 0x7f3480107b90 2024-02-21T23:40:51.070 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:47.979+0000 7f3485781700 1 -- 172.21.15.90:0/3244113781 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34800623a0 con 0x7f3480107b90 2024-02-21T23:40:51.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:47.981+0000 7f3467fff700 1 -- 172.21.15.90:0/3244113781 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f347401abd0 con 0x7f3480107b90 2024-02-21T23:40:51.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:48.048+0000 7f3485781700 1 -- 172.21.15.90:0/3244113781 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f348010a890 con 0x7f3480107b90 2024-02-21T23:40:51.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:48.050+0000 7f3467fff700 1 -- 172.21.15.90:0/3244113781 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f347401ad70 con 0x7f3480107b90 2024-02-21T23:40:51.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:48.051+0000 7f3485781700 1 -- 172.21.15.90:0/3244113781 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3480107b90 msgr2=0x7f348019a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:40:51.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:48.051+0000 7f3485781700 1 --2- 172.21.15.90:0/3244113781 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3480107b90 0x7f348019a300 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f3474011710 tx=0x7f3474011ad0 comp rx=0 tx=0).stop 2024-02-21T23:40:51.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:48.051+0000 7f3485781700 1 -- 172.21.15.90:0/3244113781 shutdown_connections 2024-02-21T23:40:51.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:48.051+0000 7f3485781700 1 --2- 172.21.15.90:0/3244113781 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3480107b90 0x7f348019a300 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:40:51.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:48.051+0000 7f3485781700 1 -- 172.21.15.90:0/3244113781 >> 172.21.15.90:0/3244113781 conn(0x7f3480101a70 msgr2=0x7f3480102660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:40:51.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:48.051+0000 7f3485781700 1 -- 172.21.15.90:0/3244113781 shutdown_connections 2024-02-21T23:40:51.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:48.051+0000 7f3485781700 1 -- 172.21.15.90:0/3244113781 wait complete. 2024-02-21T23:40:51.071 INFO:teuthology.orchestra.run.smithi090.stdout:Wrote config to /etc/ceph/ceph.conf 2024-02-21T23:40:51.339 INFO:teuthology.orchestra.run.smithi090.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-21T23:40:51.340 INFO:teuthology.orchestra.run.smithi090.stdout:Creating mgr... 2024-02-21T23:40:51.340 INFO:teuthology.orchestra.run.smithi090.stdout:Verifying port 0.0.0.0:9283 ... 2024-02-21T23:40:51.340 INFO:teuthology.orchestra.run.smithi090.stdout:Verifying port 0.0.0.0:8765 ... 2024-02-21T23:40:52.633 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-7ed81218-d112-11ee-95bd-87774f69a715@mgr.x 2024-02-21T23:40:52.633 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Failed to reset failed state of unit ceph-7ed81218-d112-11ee-95bd-87774f69a715@mgr.x.service: Unit ceph-7ed81218-d112-11ee-95bd-87774f69a715@mgr.x.service not loaded. 2024-02-21T23:40:52.724 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-7ed81218-d112-11ee-95bd-87774f69a715.target.wants/ceph-7ed81218-d112-11ee-95bd-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-7ed81218-d112-11ee-95bd-87774f69a715@.service. 2024-02-21T23:40:52.978 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:40:52 smithi090 systemd[1]: Starting Ceph mgr.x for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:40:54.180 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-21T23:40:54.180 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout disabled 2024-02-21T23:40:54.189 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-21T23:40:54.189 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout inactive 2024-02-21T23:40:54.189 INFO:teuthology.orchestra.run.smithi090.stdout:firewalld.service is not enabled 2024-02-21T23:40:54.189 INFO:teuthology.orchestra.run.smithi090.stdout:Not possible to enable service . firewalld.service is not available 2024-02-21T23:40:54.199 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-21T23:40:54.199 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout disabled 2024-02-21T23:40:54.207 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-21T23:40:54.207 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout inactive 2024-02-21T23:40:54.207 INFO:teuthology.orchestra.run.smithi090.stdout:firewalld.service is not enabled 2024-02-21T23:40:54.207 INFO:teuthology.orchestra.run.smithi090.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-02-21T23:40:54.207 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for mgr to start... 2024-02-21T23:40:54.207 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for mgr... 2024-02-21T23:40:54.478 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:40:54 smithi090 bash[43618]: 68d62e70c1b9d8109336f25cd389f7b2294b7a1fe2c1476193757f5fa26bfa63 2024-02-21T23:40:54.478 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:40:54 smithi090 systemd[1]: Started Ceph mgr.x for 7ed81218-d112-11ee-95bd-87774f69a715. 2024-02-21T23:40:54.478 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:40:54 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:40:54.410+0000 7f3c0529d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-21T23:40:54.977 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:40:54 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:40:54.502+0000 7f3c0529d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-21T23:40:54.977 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:40:54 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:40:54.589+0000 7f3c0529d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-21T23:40:54.978 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:40:54 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:40:54.673+0000 7f3c0529d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-21T23:40:55.549 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:40:55 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:40:55.310+0000 7f3c0529d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-21T23:40:55.892 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:40:55 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:40:55.564+0000 7f3c0529d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-21T23:40:55.892 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:40:55 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:40:55.649+0000 7f3c0529d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-21T23:40:55.892 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:40:55 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:40:55.762+0000 7f3c0529d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-21T23:40:56.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:55 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/1139799048' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-21T23:40:56.228 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:40:55 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:40:55.926+0000 7f3c0529d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-21T23:40:56.228 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:40:56 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:40:56.054+0000 7f3c0529d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-21T23:40:56.228 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:40:56 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:40:56.139+0000 7f3c0529d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-21T23:40:56.743 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-02-21T23:40:56.746 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout { 2024-02-21T23:40:56.746 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "fsid": "7ed81218-d112-11ee-95bd-87774f69a715", 2024-02-21T23:40:56.747 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "health": { 2024-02-21T23:40:56.747 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-21T23:40:56.747 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-21T23:40:56.747 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-21T23:40:56.747 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-02-21T23:40:56.747 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-21T23:40:56.747 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-21T23:40:56.747 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 0 2024-02-21T23:40:56.747 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-02-21T23:40:56.747 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-21T23:40:56.747 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "a" 2024-02-21T23:40:56.748 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-02-21T23:40:56.748 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-02-21T23:40:56.748 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-21T23:40:56.748 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-21T23:40:56.748 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-02-21T23:40:56.748 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-21T23:40:56.748 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-02-21T23:40:56.748 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-21T23:40:56.748 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-21T23:40:56.748 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-21T23:40:56.748 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-21T23:40:56.748 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-21T23:40:56.757 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-21T23:40:56.757 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-21T23:40:56.757 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-21T23:40:56.757 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-02-21T23:40:56.757 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-21T23:40:56.757 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-21T23:40:56.757 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-21T23:40:56.757 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-21T23:40:56.757 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-21T23:40:56.758 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-21T23:40:56.758 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-21T23:40:56.758 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-21T23:40:56.758 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-21T23:40:56.758 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-02-21T23:40:56.758 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-21T23:40:56.758 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-21T23:40:56.758 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-21T23:40:56.758 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-21T23:40:56.758 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-02-21T23:40:56.758 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-21T23:40:56.758 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-21T23:40:56.758 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-21T23:40:56.758 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-21T23:40:56.759 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-21T23:40:56.759 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-21T23:40:56.759 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "restful" 2024-02-21T23:40:56.759 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-02-21T23:40:56.759 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-21T23:40:56.759 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-02-21T23:40:56.759 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-21T23:40:56.759 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-21T23:40:56.759 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "modified": "2024-02-21T23:40:41.786090+0000", 2024-02-21T23:40:56.759 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-21T23:40:56.759 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-02-21T23:40:56.759 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-21T23:40:56.759 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout } 2024-02-21T23:40:56.759 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.800+0000 7f42efc56700 1 Processor -- start 2024-02-21T23:40:56.760 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.800+0000 7f42efc56700 1 -- start start 2024-02-21T23:40:56.760 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.800+0000 7f42efc56700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f42e8106f80 0x7f42e8109370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:40:56.760 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.800+0000 7f42efc56700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e8079f20 con 0x7f42e8106f80 2024-02-21T23:40:56.763 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.801+0000 7f42ed9f2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f42e8106f80 0x7f42e8109370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:40:56.763 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.801+0000 7f42ed9f2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f42e8106f80 0x7f42e8109370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44806/0 (socket says 172.21.15.90:44806) 2024-02-21T23:40:56.763 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.801+0000 7f42ed9f2700 1 -- 172.21.15.90:0/1641924498 learned_addr learned my addr 172.21.15.90:0/1641924498 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:40:56.764 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.801+0000 7f42ed9f2700 1 -- 172.21.15.90:0/1641924498 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e81098b0 con 0x7f42e8106f80 2024-02-21T23:40:56.764 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.801+0000 7f42ed9f2700 1 --2- 172.21.15.90:0/1641924498 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f42e8106f80 0x7f42e8109370 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f42e40098a0 tx=0x7f42e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb375c5572433e71 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:40:56.764 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.801+0000 7f42ec9f0700 1 -- 172.21.15.90:0/1641924498 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f42e4004160 con 0x7f42e8106f80 2024-02-21T23:40:56.764 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.801+0000 7f42ec9f0700 1 -- 172.21.15.90:0/1641924498 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f42e40042c0 con 0x7f42e8106f80 2024-02-21T23:40:56.764 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.801+0000 7f42ec9f0700 1 -- 172.21.15.90:0/1641924498 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f42e4011a00 con 0x7f42e8106f80 2024-02-21T23:40:56.764 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.802+0000 7f42efc56700 1 -- 172.21.15.90:0/1641924498 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f42e8106f80 msgr2=0x7f42e8109370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:40:56.764 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.802+0000 7f42efc56700 1 --2- 172.21.15.90:0/1641924498 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f42e8106f80 0x7f42e8109370 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f42e40098a0 tx=0x7f42e4009bb0 comp rx=0 tx=0).stop 2024-02-21T23:40:56.764 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.802+0000 7f42efc56700 1 -- 172.21.15.90:0/1641924498 shutdown_connections 2024-02-21T23:40:56.764 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.802+0000 7f42efc56700 1 --2- 172.21.15.90:0/1641924498 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f42e8106f80 0x7f42e8109370 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:40:56.764 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.802+0000 7f42efc56700 1 -- 172.21.15.90:0/1641924498 >> 172.21.15.90:0/1641924498 conn(0x7f42e8100970 msgr2=0x7f42e8102d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:40:56.764 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.802+0000 7f42efc56700 1 -- 172.21.15.90:0/1641924498 shutdown_connections 2024-02-21T23:40:56.765 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.802+0000 7f42efc56700 1 -- 172.21.15.90:0/1641924498 wait complete. 2024-02-21T23:40:56.765 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.802+0000 7f42efc56700 1 Processor -- start 2024-02-21T23:40:56.765 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.802+0000 7f42efc56700 1 -- start start 2024-02-21T23:40:56.765 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.803+0000 7f42efc56700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f42e8106f80 0x7f42e81067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:40:56.765 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.803+0000 7f42efc56700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e8106cf0 con 0x7f42e8106f80 2024-02-21T23:40:56.765 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.803+0000 7f42ed9f2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f42e8106f80 0x7f42e81067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:40:56.765 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.803+0000 7f42ed9f2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f42e8106f80 0x7f42e81067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44818/0 (socket says 172.21.15.90:44818) 2024-02-21T23:40:56.765 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.803+0000 7f42ed9f2700 1 -- 172.21.15.90:0/1139799048 learned_addr learned my addr 172.21.15.90:0/1139799048 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:40:56.765 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.803+0000 7f42ed9f2700 1 -- 172.21.15.90:0/1139799048 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e4009550 con 0x7f42e8106f80 2024-02-21T23:40:56.766 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.803+0000 7f42ed9f2700 1 --2- 172.21.15.90:0/1139799048 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f42e8106f80 0x7f42e81067b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f42e4011a20 tx=0x7f42e4011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:40:56.766 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.803+0000 7f42deffd700 1 -- 172.21.15.90:0/1139799048 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f42e4003af0 con 0x7f42e8106f80 2024-02-21T23:40:56.766 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.803+0000 7f42deffd700 1 -- 172.21.15.90:0/1139799048 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f42e4003c50 con 0x7f42e8106f80 2024-02-21T23:40:56.766 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.803+0000 7f42deffd700 1 -- 172.21.15.90:0/1139799048 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f42e4019760 con 0x7f42e8106f80 2024-02-21T23:40:56.766 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.803+0000 7f42efc56700 1 -- 172.21.15.90:0/1139799048 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42e8104f30 con 0x7f42e8106f80 2024-02-21T23:40:56.766 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.803+0000 7f42efc56700 1 -- 172.21.15.90:0/1139799048 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42e8105430 con 0x7f42e8106f80 2024-02-21T23:40:56.766 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.803+0000 7f42deffd700 1 -- 172.21.15.90:0/1139799048 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f42e4019950 con 0x7f42e8106f80 2024-02-21T23:40:56.766 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.803+0000 7f42deffd700 1 -- 172.21.15.90:0/1139799048 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f42e400b3e0 con 0x7f42e8106f80 2024-02-21T23:40:56.766 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.804+0000 7f42efc56700 1 -- 172.21.15.90:0/1139799048 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42e80623a0 con 0x7f42e8106f80 2024-02-21T23:40:56.766 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.806+0000 7f42deffd700 1 -- 172.21.15.90:0/1139799048 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f42e402a060 con 0x7f42e8106f80 2024-02-21T23:40:56.766 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.880+0000 7f42efc56700 1 -- 172.21.15.90:0/1139799048 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f42e8105de0 con 0x7f42e8106f80 2024-02-21T23:40:56.767 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.880+0000 7f42deffd700 1 -- 172.21.15.90:0/1139799048 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f42e4019d90 con 0x7f42e8106f80 2024-02-21T23:40:56.767 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.881+0000 7f42efc56700 1 -- 172.21.15.90:0/1139799048 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f42e8106f80 msgr2=0x7f42e81067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:40:56.767 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.881+0000 7f42efc56700 1 --2- 172.21.15.90:0/1139799048 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f42e8106f80 0x7f42e81067b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f42e4011a20 tx=0x7f42e4011b00 comp rx=0 tx=0).stop 2024-02-21T23:40:56.767 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.881+0000 7f42efc56700 1 -- 172.21.15.90:0/1139799048 shutdown_connections 2024-02-21T23:40:56.767 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.881+0000 7f42efc56700 1 --2- 172.21.15.90:0/1139799048 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f42e8106f80 0x7f42e81067b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:40:56.767 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.881+0000 7f42efc56700 1 -- 172.21.15.90:0/1139799048 >> 172.21.15.90:0/1139799048 conn(0x7f42e8100970 msgr2=0x7f42e8101520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:40:56.767 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.881+0000 7f42efc56700 1 -- 172.21.15.90:0/1139799048 shutdown_connections 2024-02-21T23:40:56.767 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:55.881+0000 7f42efc56700 1 -- 172.21.15.90:0/1139799048 wait complete. 2024-02-21T23:40:56.767 INFO:teuthology.orchestra.run.smithi090.stdout:mgr not available, waiting (1/15)... 2024-02-21T23:40:56.885 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:40:56 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:40:56.510+0000 7f3c0529d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-21T23:40:57.167 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:40:56 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:40:56.900+0000 7f3c0529d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-21T23:40:57.167 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:40:57 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:40:57.091+0000 7f3c0529d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-21T23:40:57.167 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:40:57 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:40:57.183+0000 7f3c0529d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-21T23:40:58.227 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:40:57 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:40:57.936+0000 7f3c0529d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-21T23:40:58.227 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:40:58 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:40:58.032+0000 7f3c0529d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-21T23:40:58.227 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:40:58 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:40:58.179+0000 7f3c0529d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-21T23:40:59.196 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:40:58 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:40:58.805+0000 7f3c0529d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-21T23:40:59.197 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:40:59 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:40:59.002+0000 7f3c0529d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-21T23:40:59.197 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:40:59 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:40:59.212+0000 7f3c0529d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-21T23:40:59.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:40:59 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/1640885601' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-21T23:40:59.664 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-02-21T23:40:59.664 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout { 2024-02-21T23:40:59.664 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "fsid": "7ed81218-d112-11ee-95bd-87774f69a715", 2024-02-21T23:40:59.664 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "health": { 2024-02-21T23:40:59.664 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-21T23:40:59.664 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-21T23:40:59.665 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-21T23:40:59.665 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-02-21T23:40:59.665 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-21T23:40:59.665 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-21T23:40:59.665 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 0 2024-02-21T23:40:59.665 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-02-21T23:40:59.665 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-21T23:40:59.665 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "a" 2024-02-21T23:40:59.665 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-02-21T23:40:59.665 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-02-21T23:40:59.665 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-21T23:40:59.665 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-21T23:40:59.665 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-02-21T23:40:59.665 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-21T23:40:59.665 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-02-21T23:40:59.665 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-21T23:40:59.666 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-21T23:40:59.666 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-21T23:40:59.666 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-21T23:40:59.666 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-21T23:40:59.666 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-21T23:40:59.666 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-21T23:40:59.666 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-21T23:40:59.666 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-02-21T23:40:59.666 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-21T23:40:59.666 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-21T23:40:59.666 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-21T23:40:59.666 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-21T23:40:59.666 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-21T23:40:59.666 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-21T23:40:59.667 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-21T23:40:59.667 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-21T23:40:59.667 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-21T23:40:59.667 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-02-21T23:40:59.667 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-21T23:40:59.667 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-21T23:40:59.667 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-21T23:40:59.667 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-21T23:40:59.667 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-02-21T23:40:59.667 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-21T23:40:59.667 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-21T23:40:59.667 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-21T23:40:59.667 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-21T23:40:59.667 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-21T23:40:59.668 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-21T23:40:59.668 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "restful" 2024-02-21T23:40:59.668 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-02-21T23:40:59.668 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-21T23:40:59.668 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-02-21T23:40:59.668 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-21T23:40:59.668 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-21T23:40:59.668 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "modified": "2024-02-21T23:40:41.786090+0000", 2024-02-21T23:40:59.668 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-21T23:40:59.668 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-02-21T23:40:59.668 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-21T23:40:59.668 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout } 2024-02-21T23:40:59.668 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.159+0000 7fb9c0660700 1 Processor -- start 2024-02-21T23:40:59.669 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.159+0000 7fb9c0660700 1 -- start start 2024-02-21T23:40:59.669 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.159+0000 7fb9c0660700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9b8106f00 0x7fb9b81092f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:40:59.669 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.159+0000 7fb9c0660700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b8109830 con 0x7fb9b8106f00 2024-02-21T23:40:59.669 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.159+0000 7fb9be3fc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9b8106f00 0x7fb9b81092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:40:59.669 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.159+0000 7fb9be3fc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9b8106f00 0x7fb9b81092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59988/0 (socket says 172.21.15.90:59988) 2024-02-21T23:40:59.669 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.159+0000 7fb9be3fc700 1 -- 172.21.15.90:0/914839711 learned_addr learned my addr 172.21.15.90:0/914839711 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:40:59.669 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.160+0000 7fb9be3fc700 1 -- 172.21.15.90:0/914839711 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b8109970 con 0x7fb9b8106f00 2024-02-21T23:40:59.669 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.160+0000 7fb9be3fc700 1 --2- 172.21.15.90:0/914839711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9b8106f00 0x7fb9b81092f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb9b40098a0 tx=0x7fb9b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=374160a26d59d273 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:40:59.669 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.160+0000 7fb9bd3fa700 1 -- 172.21.15.90:0/914839711 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb9b4004160 con 0x7fb9b8106f00 2024-02-21T23:40:59.669 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.160+0000 7fb9bd3fa700 1 -- 172.21.15.90:0/914839711 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb9b40042c0 con 0x7fb9b8106f00 2024-02-21T23:40:59.669 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.160+0000 7fb9bd3fa700 1 -- 172.21.15.90:0/914839711 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb9b4011a00 con 0x7fb9b8106f00 2024-02-21T23:40:59.669 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.161+0000 7fb9c0660700 1 -- 172.21.15.90:0/914839711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9b8106f00 msgr2=0x7fb9b81092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:40:59.669 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.161+0000 7fb9c0660700 1 --2- 172.21.15.90:0/914839711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9b8106f00 0x7fb9b81092f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb9b40098a0 tx=0x7fb9b4009bb0 comp rx=0 tx=0).stop 2024-02-21T23:40:59.670 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.161+0000 7fb9c0660700 1 -- 172.21.15.90:0/914839711 shutdown_connections 2024-02-21T23:40:59.670 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.161+0000 7fb9c0660700 1 --2- 172.21.15.90:0/914839711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9b8106f00 0x7fb9b81092f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:40:59.670 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.161+0000 7fb9c0660700 1 -- 172.21.15.90:0/914839711 >> 172.21.15.90:0/914839711 conn(0x7fb9b8100970 msgr2=0x7fb9b8102d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:40:59.670 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.161+0000 7fb9c0660700 1 -- 172.21.15.90:0/914839711 shutdown_connections 2024-02-21T23:40:59.670 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.161+0000 7fb9c0660700 1 -- 172.21.15.90:0/914839711 wait complete. 2024-02-21T23:40:59.670 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.161+0000 7fb9c0660700 1 Processor -- start 2024-02-21T23:40:59.670 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.162+0000 7fb9c0660700 1 -- start start 2024-02-21T23:40:59.670 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.162+0000 7fb9c0660700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9b8106f00 0x7fb9b819a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:40:59.670 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.162+0000 7fb9c0660700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b819a860 con 0x7fb9b8106f00 2024-02-21T23:40:59.670 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.162+0000 7fb9be3fc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9b8106f00 0x7fb9b819a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:40:59.670 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.162+0000 7fb9be3fc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9b8106f00 0x7fb9b819a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59994/0 (socket says 172.21.15.90:59994) 2024-02-21T23:40:59.670 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.162+0000 7fb9be3fc700 1 -- 172.21.15.90:0/1640885601 learned_addr learned my addr 172.21.15.90:0/1640885601 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:40:59.670 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.162+0000 7fb9be3fc700 1 -- 172.21.15.90:0/1640885601 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b4009550 con 0x7fb9b8106f00 2024-02-21T23:40:59.671 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.162+0000 7fb9be3fc700 1 --2- 172.21.15.90:0/1640885601 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9b8106f00 0x7fb9b819a320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4009130 tx=0x7fb9b4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:40:59.671 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.162+0000 7fb9af7fe700 1 -- 172.21.15.90:0/1640885601 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb9b40038b0 con 0x7fb9b8106f00 2024-02-21T23:40:59.671 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.163+0000 7fb9af7fe700 1 -- 172.21.15.90:0/1640885601 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb9b4003a10 con 0x7fb9b8106f00 2024-02-21T23:40:59.671 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.163+0000 7fb9af7fe700 1 -- 172.21.15.90:0/1640885601 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb9b4019550 con 0x7fb9b8106f00 2024-02-21T23:40:59.671 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.163+0000 7fb9c0660700 1 -- 172.21.15.90:0/1640885601 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9b819aa60 con 0x7fb9b8106f00 2024-02-21T23:40:59.671 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.163+0000 7fb9c0660700 1 -- 172.21.15.90:0/1640885601 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9b819af00 con 0x7fb9b8106f00 2024-02-21T23:40:59.671 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.163+0000 7fb9af7fe700 1 -- 172.21.15.90:0/1640885601 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb9b4019ac0 con 0x7fb9b8106f00 2024-02-21T23:40:59.671 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.163+0000 7fb9af7fe700 1 -- 172.21.15.90:0/1640885601 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb9b402dde0 con 0x7fb9b8106f00 2024-02-21T23:40:59.671 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.164+0000 7fb9c0660700 1 -- 172.21.15.90:0/1640885601 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9b80623a0 con 0x7fb9b8106f00 2024-02-21T23:40:59.671 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.166+0000 7fb9af7fe700 1 -- 172.21.15.90:0/1640885601 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fb9b4003b80 con 0x7fb9b8106f00 2024-02-21T23:40:59.671 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.238+0000 7fb9c0660700 1 -- 172.21.15.90:0/1640885601 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb9b819d880 con 0x7fb9b8106f00 2024-02-21T23:40:59.671 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.239+0000 7fb9af7fe700 1 -- 172.21.15.90:0/1640885601 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7fb9b401a960 con 0x7fb9b8106f00 2024-02-21T23:40:59.671 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.240+0000 7fb9c0660700 1 -- 172.21.15.90:0/1640885601 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9b8106f00 msgr2=0x7fb9b819a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:40:59.671 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.240+0000 7fb9c0660700 1 --2- 172.21.15.90:0/1640885601 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9b8106f00 0x7fb9b819a320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4009130 tx=0x7fb9b4011ad0 comp rx=0 tx=0).stop 2024-02-21T23:40:59.672 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.240+0000 7fb9c0660700 1 -- 172.21.15.90:0/1640885601 shutdown_connections 2024-02-21T23:40:59.672 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.240+0000 7fb9c0660700 1 --2- 172.21.15.90:0/1640885601 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9b8106f00 0x7fb9b819a320 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:40:59.672 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.240+0000 7fb9c0660700 1 -- 172.21.15.90:0/1640885601 >> 172.21.15.90:0/1640885601 conn(0x7fb9b8100970 msgr2=0x7fb9b8101540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:40:59.672 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.240+0000 7fb9c0660700 1 -- 172.21.15.90:0/1640885601 shutdown_connections 2024-02-21T23:40:59.672 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:40:59.240+0000 7fb9c0660700 1 -- 172.21.15.90:0/1640885601 wait complete. 2024-02-21T23:40:59.672 INFO:teuthology.orchestra.run.smithi090.stdout:mgr not available, waiting (2/15)... 2024-02-21T23:41:00.426 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:00 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:00.161+0000 7f3c0529d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-21T23:41:00.702 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:00 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:00.441+0000 7f3c0529d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-21T23:41:00.702 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:00 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:00.528+0000 7f3c0529d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-21T23:41:00.977 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:00 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:00.717+0000 7f3c0529d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-21T23:41:00.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:00 smithi090 ceph-mon[43385]: Activating manager daemon x 2024-02-21T23:41:00.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:00 smithi090 ceph-mon[43385]: mgrmap e2: x(active, starting, since 0.00281376s) 2024-02-21T23:41:00.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:00 smithi090 ceph-mon[43385]: from='mgr.14100 172.21.15.90:0/2956486076' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-21T23:41:00.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:00 smithi090 ceph-mon[43385]: from='mgr.14100 172.21.15.90:0/2956486076' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-21T23:41:00.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:00 smithi090 ceph-mon[43385]: from='mgr.14100 172.21.15.90:0/2956486076' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-21T23:41:00.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:00 smithi090 ceph-mon[43385]: from='mgr.14100 172.21.15.90:0/2956486076' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-21T23:41:00.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:00 smithi090 ceph-mon[43385]: from='mgr.14100 172.21.15.90:0/2956486076' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-21T23:41:00.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:00 smithi090 ceph-mon[43385]: Manager daemon x is now available 2024-02-21T23:41:00.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:00 smithi090 ceph-mon[43385]: from='mgr.14100 172.21.15.90:0/2956486076' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-21T23:41:00.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:00 smithi090 ceph-mon[43385]: from='mgr.14100 172.21.15.90:0/2956486076' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-21T23:41:00.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:00 smithi090 ceph-mon[43385]: from='mgr.14100 172.21.15.90:0/2956486076' entity='mgr.x' 2024-02-21T23:41:00.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:00 smithi090 ceph-mon[43385]: from='mgr.14100 172.21.15.90:0/2956486076' entity='mgr.x' 2024-02-21T23:41:00.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:00 smithi090 ceph-mon[43385]: from='mgr.14100 172.21.15.90:0/2956486076' entity='mgr.x' 2024-02-21T23:41:02.771 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-02-21T23:41:02.771 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout { 2024-02-21T23:41:02.771 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "fsid": "7ed81218-d112-11ee-95bd-87774f69a715", 2024-02-21T23:41:02.771 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "health": { 2024-02-21T23:41:02.771 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-21T23:41:02.771 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-21T23:41:02.771 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-21T23:41:02.771 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-02-21T23:41:02.771 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-21T23:41:02.771 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-21T23:41:02.771 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 0 2024-02-21T23:41:02.772 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-02-21T23:41:02.772 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-21T23:41:02.772 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "a" 2024-02-21T23:41:02.772 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-02-21T23:41:02.772 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum_age": 15, 2024-02-21T23:41:02.772 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-21T23:41:02.772 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-21T23:41:02.772 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-02-21T23:41:02.772 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-21T23:41:02.772 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-02-21T23:41:02.772 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-21T23:41:02.772 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-21T23:41:02.772 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-21T23:41:02.772 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-21T23:41:02.773 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-21T23:41:02.773 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-21T23:41:02.773 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-21T23:41:02.773 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-21T23:41:02.773 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-02-21T23:41:02.773 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-21T23:41:02.773 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-21T23:41:02.773 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-21T23:41:02.773 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-21T23:41:02.773 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-21T23:41:02.773 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-21T23:41:02.773 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-21T23:41:02.774 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-21T23:41:02.774 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-21T23:41:02.774 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-02-21T23:41:02.774 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-21T23:41:02.774 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-21T23:41:02.774 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-21T23:41:02.774 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-21T23:41:02.774 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-02-21T23:41:02.774 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-21T23:41:02.774 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-21T23:41:02.774 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-21T23:41:02.774 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-21T23:41:02.774 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-21T23:41:02.774 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-21T23:41:02.775 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "restful" 2024-02-21T23:41:02.775 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-02-21T23:41:02.775 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-21T23:41:02.775 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-02-21T23:41:02.775 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-21T23:41:02.775 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-21T23:41:02.775 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "modified": "2024-02-21T23:40:41.786090+0000", 2024-02-21T23:41:02.775 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-21T23:41:02.775 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-02-21T23:41:02.775 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-21T23:41:02.775 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout } 2024-02-21T23:41:02.775 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.055+0000 7f9386f69700 1 Processor -- start 2024-02-21T23:41:02.776 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.055+0000 7f9386f69700 1 -- start start 2024-02-21T23:41:02.776 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.055+0000 7f9386f69700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9380106a50 0x7f9380106e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:02.776 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.055+0000 7f9386f69700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9380107440 con 0x7f9380106a50 2024-02-21T23:41:02.776 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.055+0000 7f9384d05700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9380106a50 0x7f9380106e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:02.776 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.055+0000 7f9384d05700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9380106a50 0x7f9380106e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60064/0 (socket says 172.21.15.90:60064) 2024-02-21T23:41:02.776 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.055+0000 7f9384d05700 1 -- 172.21.15.90:0/4023622539 learned_addr learned my addr 172.21.15.90:0/4023622539 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:02.776 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.056+0000 7f9384d05700 1 -- 172.21.15.90:0/4023622539 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9380105510 con 0x7f9380106a50 2024-02-21T23:41:02.776 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.056+0000 7f9384d05700 1 --2- 172.21.15.90:0/4023622539 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9380106a50 0x7f9380106e70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f93740098a0 tx=0x7f9374009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18404740e70b3651 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:02.776 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.056+0000 7f937f7fe700 1 -- 172.21.15.90:0/4023622539 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9374004160 con 0x7f9380106a50 2024-02-21T23:41:02.776 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.056+0000 7f937f7fe700 1 -- 172.21.15.90:0/4023622539 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f93740042c0 con 0x7f9380106a50 2024-02-21T23:41:02.776 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.056+0000 7f937f7fe700 1 -- 172.21.15.90:0/4023622539 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9374011a00 con 0x7f9380106a50 2024-02-21T23:41:02.776 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.056+0000 7f9386f69700 1 -- 172.21.15.90:0/4023622539 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9380106a50 msgr2=0x7f9380106e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:02.776 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.056+0000 7f9386f69700 1 --2- 172.21.15.90:0/4023622539 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9380106a50 0x7f9380106e70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f93740098a0 tx=0x7f9374009bb0 comp rx=0 tx=0).stop 2024-02-21T23:41:02.777 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.056+0000 7f9386f69700 1 -- 172.21.15.90:0/4023622539 shutdown_connections 2024-02-21T23:41:02.777 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.056+0000 7f9386f69700 1 --2- 172.21.15.90:0/4023622539 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9380106a50 0x7f9380106e70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:02.777 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.057+0000 7f9386f69700 1 -- 172.21.15.90:0/4023622539 >> 172.21.15.90:0/4023622539 conn(0x7f93801000a0 msgr2=0x7f9380102500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:02.777 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.057+0000 7f9386f69700 1 -- 172.21.15.90:0/4023622539 shutdown_connections 2024-02-21T23:41:02.777 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.057+0000 7f9386f69700 1 -- 172.21.15.90:0/4023622539 wait complete. 2024-02-21T23:41:02.777 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.057+0000 7f9386f69700 1 Processor -- start 2024-02-21T23:41:02.777 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.057+0000 7f9386f69700 1 -- start start 2024-02-21T23:41:02.777 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.057+0000 7f9386f69700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9380106a50 0x7f9380104e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:02.777 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.057+0000 7f9386f69700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9380105390 con 0x7f9380106a50 2024-02-21T23:41:02.777 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.057+0000 7f9384d05700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9380106a50 0x7f9380104e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:02.777 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.057+0000 7f9384d05700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9380106a50 0x7f9380104e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60068/0 (socket says 172.21.15.90:60068) 2024-02-21T23:41:02.777 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.057+0000 7f9384d05700 1 -- 172.21.15.90:0/2923367694 learned_addr learned my addr 172.21.15.90:0/2923367694 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:02.777 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.058+0000 7f9384d05700 1 -- 172.21.15.90:0/2923367694 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9374009550 con 0x7f9380106a50 2024-02-21T23:41:02.777 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.058+0000 7f9384d05700 1 --2- 172.21.15.90:0/2923367694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9380106a50 0x7f9380104e50 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9374009130 tx=0x7f9374011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:02.778 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.058+0000 7f937dffb700 1 -- 172.21.15.90:0/2923367694 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f93740038b0 con 0x7f9380106a50 2024-02-21T23:41:02.778 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.058+0000 7f937dffb700 1 -- 172.21.15.90:0/2923367694 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9374003a10 con 0x7f9380106a50 2024-02-21T23:41:02.778 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.058+0000 7f937dffb700 1 -- 172.21.15.90:0/2923367694 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9374019550 con 0x7f9380106a50 2024-02-21T23:41:02.778 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.058+0000 7f9386f69700 1 -- 172.21.15.90:0/2923367694 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f938019d010 con 0x7f9380106a50 2024-02-21T23:41:02.778 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.058+0000 7f9386f69700 1 -- 172.21.15.90:0/2923367694 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f938019d4b0 con 0x7f9380106a50 2024-02-21T23:41:02.778 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.059+0000 7f9386f69700 1 -- 172.21.15.90:0/2923367694 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93800623a0 con 0x7f9380106a50 2024-02-21T23:41:02.778 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.060+0000 7f937dffb700 1 -- 172.21.15.90:0/2923367694 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 3) v1 ==== 45420+0+0 (secure 0 0 0) 0x7f937401a960 con 0x7f9380106a50 2024-02-21T23:41:02.778 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.060+0000 7f937dffb700 1 --2- 172.21.15.90:0/2923367694 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7f9370038520 0x7f937003a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:02.778 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.060+0000 7f937dffb700 1 -- 172.21.15.90:0/2923367694 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f937404c650 con 0x7f9380106a50 2024-02-21T23:41:02.778 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.060+0000 7f937ffff700 1 --2- 172.21.15.90:0/2923367694 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7f9370038520 0x7f937003a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:02.778 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.060+0000 7f937ffff700 1 --2- 172.21.15.90:0/2923367694 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7f9370038520 0x7f937003a9e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f936c006fd0 tx=0x7f936c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:02.778 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.065+0000 7f937dffb700 1 -- 172.21.15.90:0/2923367694 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f9374003b80 con 0x7f9380106a50 2024-02-21T23:41:02.778 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.333+0000 7f9386f69700 1 -- 172.21.15.90:0/2923367694 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f9380109770 con 0x7f9380106a50 2024-02-21T23:41:02.778 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.333+0000 7f937dffb700 1 -- 172.21.15.90:0/2923367694 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f937404b090 con 0x7f9380106a50 2024-02-21T23:41:02.779 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.335+0000 7f9386f69700 1 -- 172.21.15.90:0/2923367694 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7f9370038520 msgr2=0x7f937003a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:02.779 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.335+0000 7f9386f69700 1 --2- 172.21.15.90:0/2923367694 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7f9370038520 0x7f937003a9e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f936c006fd0 tx=0x7f936c006e40 comp rx=0 tx=0).stop 2024-02-21T23:41:02.779 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.335+0000 7f9386f69700 1 -- 172.21.15.90:0/2923367694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9380106a50 msgr2=0x7f9380104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:02.779 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.335+0000 7f9386f69700 1 --2- 172.21.15.90:0/2923367694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9380106a50 0x7f9380104e50 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9374009130 tx=0x7f9374011ad0 comp rx=0 tx=0).stop 2024-02-21T23:41:02.779 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.335+0000 7f9386f69700 1 -- 172.21.15.90:0/2923367694 shutdown_connections 2024-02-21T23:41:02.779 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.335+0000 7f9386f69700 1 --2- 172.21.15.90:0/2923367694 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7f9370038520 0x7f937003a9e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:02.779 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.335+0000 7f9386f69700 1 --2- 172.21.15.90:0/2923367694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9380106a50 0x7f9380104e50 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:02.779 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.335+0000 7f9386f69700 1 -- 172.21.15.90:0/2923367694 >> 172.21.15.90:0/2923367694 conn(0x7f93801000a0 msgr2=0x7f9380100ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:02.779 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.336+0000 7f9386f69700 1 -- 172.21.15.90:0/2923367694 shutdown_connections 2024-02-21T23:41:02.779 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:02.336+0000 7f9386f69700 1 -- 172.21.15.90:0/2923367694 wait complete. 2024-02-21T23:41:02.779 INFO:teuthology.orchestra.run.smithi090.stdout:mgr is available 2024-02-21T23:41:02.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:02 smithi090 ceph-mon[43385]: mgrmap e3: x(active, since 1.00872s) 2024-02-21T23:41:02.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:02 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/2923367694' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-21T23:41:03.808 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-02-21T23:41:03.809 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout [global] 2024-02-21T23:41:03.809 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout fsid = 7ed81218-d112-11ee-95bd-87774f69a715 2024-02-21T23:41:03.809 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-21T23:41:03.809 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-21T23:41:03.809 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-21T23:41:03.809 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-21T23:41:03.809 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-02-21T23:41:03.809 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-21T23:41:03.809 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-21T23:41:03.809 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-02-21T23:41:03.809 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout [osd] 2024-02-21T23:41:03.809 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-02-21T23:41:03.809 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-02-21T23:41:03.810 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-02-21T23:41:03.810 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-21T23:41:03.810 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-21T23:41:03.810 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.154+0000 7f499a898700 1 Processor -- start 2024-02-21T23:41:03.810 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.154+0000 7f499a898700 1 -- start start 2024-02-21T23:41:03.810 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.155+0000 7f499a898700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4994106fa0 0x7f4994109390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:03.810 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.155+0000 7f499a898700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4994079e60 con 0x7f4994106fa0 2024-02-21T23:41:03.810 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.155+0000 7f4993fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4994106fa0 0x7f4994109390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:03.810 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.155+0000 7f4993fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4994106fa0 0x7f4994109390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60072/0 (socket says 172.21.15.90:60072) 2024-02-21T23:41:03.810 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.155+0000 7f4993fff700 1 -- 172.21.15.90:0/2610428500 learned_addr learned my addr 172.21.15.90:0/2610428500 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:03.810 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.155+0000 7f4993fff700 1 -- 172.21.15.90:0/2610428500 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49941098d0 con 0x7f4994106fa0 2024-02-21T23:41:03.810 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.155+0000 7f4993fff700 1 --2- 172.21.15.90:0/2610428500 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4994106fa0 0x7f4994109390 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f49840098a0 tx=0x7f4984009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6994c893a7e4b55 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:03.810 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.155+0000 7f4992ffd700 1 -- 172.21.15.90:0/2610428500 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4984004160 con 0x7f4994106fa0 2024-02-21T23:41:03.811 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.155+0000 7f4992ffd700 1 -- 172.21.15.90:0/2610428500 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f49840042c0 con 0x7f4994106fa0 2024-02-21T23:41:03.811 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.155+0000 7f4992ffd700 1 -- 172.21.15.90:0/2610428500 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4984011a00 con 0x7f4994106fa0 2024-02-21T23:41:03.811 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.156+0000 7f499a898700 1 -- 172.21.15.90:0/2610428500 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4994106fa0 msgr2=0x7f4994109390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:03.811 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.156+0000 7f499a898700 1 --2- 172.21.15.90:0/2610428500 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4994106fa0 0x7f4994109390 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f49840098a0 tx=0x7f4984009bb0 comp rx=0 tx=0).stop 2024-02-21T23:41:03.811 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.156+0000 7f499a898700 1 -- 172.21.15.90:0/2610428500 shutdown_connections 2024-02-21T23:41:03.811 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.156+0000 7f499a898700 1 --2- 172.21.15.90:0/2610428500 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4994106fa0 0x7f4994109390 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:03.811 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.156+0000 7f499a898700 1 -- 172.21.15.90:0/2610428500 >> 172.21.15.90:0/2610428500 conn(0x7f4994100a30 msgr2=0x7f4994102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:03.811 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.156+0000 7f499a898700 1 -- 172.21.15.90:0/2610428500 shutdown_connections 2024-02-21T23:41:03.811 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.156+0000 7f499a898700 1 -- 172.21.15.90:0/2610428500 wait complete. 2024-02-21T23:41:03.811 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.157+0000 7f499a898700 1 Processor -- start 2024-02-21T23:41:03.811 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.157+0000 7f499a898700 1 -- start start 2024-02-21T23:41:03.811 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.157+0000 7f499a898700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4994106fa0 0x7f499419a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:03.811 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.157+0000 7f499a898700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499419a680 con 0x7f4994106fa0 2024-02-21T23:41:03.811 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.157+0000 7f4993fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4994106fa0 0x7f499419a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:03.811 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.157+0000 7f4993fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4994106fa0 0x7f499419a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60078/0 (socket says 172.21.15.90:60078) 2024-02-21T23:41:03.812 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.157+0000 7f4993fff700 1 -- 172.21.15.90:0/3216798985 learned_addr learned my addr 172.21.15.90:0/3216798985 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:03.812 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.157+0000 7f4993fff700 1 -- 172.21.15.90:0/3216798985 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4984009550 con 0x7f4994106fa0 2024-02-21T23:41:03.812 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.157+0000 7f4993fff700 1 --2- 172.21.15.90:0/3216798985 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4994106fa0 0x7f499419a140 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4984009130 tx=0x7f4984011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:03.812 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.158+0000 7f49917fa700 1 -- 172.21.15.90:0/3216798985 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4984003b70 con 0x7f4994106fa0 2024-02-21T23:41:03.812 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.158+0000 7f49917fa700 1 -- 172.21.15.90:0/3216798985 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4984003cd0 con 0x7f4994106fa0 2024-02-21T23:41:03.812 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.158+0000 7f499a898700 1 -- 172.21.15.90:0/3216798985 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f499419a880 con 0x7f4994106fa0 2024-02-21T23:41:03.812 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.158+0000 7f49917fa700 1 -- 172.21.15.90:0/3216798985 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f49840197d0 con 0x7f4994106fa0 2024-02-21T23:41:03.812 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.158+0000 7f499a898700 1 -- 172.21.15.90:0/3216798985 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f499419ad20 con 0x7f4994106fa0 2024-02-21T23:41:03.812 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.158+0000 7f49917fa700 1 -- 172.21.15.90:0/3216798985 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 3) v1 ==== 45420+0+0 (secure 0 0 0) 0x7f4984019a10 con 0x7f4994106fa0 2024-02-21T23:41:03.812 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.158+0000 7f499a898700 1 -- 172.21.15.90:0/3216798985 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f499410a600 con 0x7f4994106fa0 2024-02-21T23:41:03.812 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.159+0000 7f49917fa700 1 --2- 172.21.15.90:0/3216798985 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7f4980038580 0x7f498003aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:03.812 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.159+0000 7f49917fa700 1 -- 172.21.15.90:0/3216798985 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f498404c690 con 0x7f4994106fa0 2024-02-21T23:41:03.812 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.159+0000 7f49937fe700 1 --2- 172.21.15.90:0/3216798985 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7f4980038580 0x7f498003aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:03.813 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.159+0000 7f49937fe700 1 --2- 172.21.15.90:0/3216798985 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7f4980038580 0x7f498003aa40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f497c006fd0 tx=0x7f497c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:03.813 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.163+0000 7f49917fa700 1 -- 172.21.15.90:0/3216798985 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f498400b900 con 0x7f4994106fa0 2024-02-21T23:41:03.813 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.354+0000 7f499a898700 1 -- 172.21.15.90:0/3216798985 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f499410a7f0 con 0x7f4994106fa0 2024-02-21T23:41:03.813 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.355+0000 7f49917fa700 1 -- 172.21.15.90:0/3216798985 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f4984018350 con 0x7f4994106fa0 2024-02-21T23:41:03.813 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.357+0000 7f499a898700 1 -- 172.21.15.90:0/3216798985 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7f4980038580 msgr2=0x7f498003aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:03.813 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.357+0000 7f499a898700 1 --2- 172.21.15.90:0/3216798985 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7f4980038580 0x7f498003aa40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f497c006fd0 tx=0x7f497c006e40 comp rx=0 tx=0).stop 2024-02-21T23:41:03.813 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.357+0000 7f499a898700 1 -- 172.21.15.90:0/3216798985 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4994106fa0 msgr2=0x7f499419a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:03.813 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.357+0000 7f499a898700 1 --2- 172.21.15.90:0/3216798985 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4994106fa0 0x7f499419a140 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4984009130 tx=0x7f4984011ad0 comp rx=0 tx=0).stop 2024-02-21T23:41:03.813 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.357+0000 7f499a898700 1 -- 172.21.15.90:0/3216798985 shutdown_connections 2024-02-21T23:41:03.813 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.357+0000 7f499a898700 1 --2- 172.21.15.90:0/3216798985 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7f4980038580 0x7f498003aa40 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:03.813 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.357+0000 7f499a898700 1 --2- 172.21.15.90:0/3216798985 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4994106fa0 0x7f499419a140 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:03.813 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.357+0000 7f499a898700 1 -- 172.21.15.90:0/3216798985 >> 172.21.15.90:0/3216798985 conn(0x7f4994100a30 msgr2=0x7f4994101500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:03.813 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.357+0000 7f499a898700 1 -- 172.21.15.90:0/3216798985 shutdown_connections 2024-02-21T23:41:03.813 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:03.357+0000 7f499a898700 1 -- 172.21.15.90:0/3216798985 wait complete. 2024-02-21T23:41:03.814 INFO:teuthology.orchestra.run.smithi090.stdout:Enabling cephadm module... 2024-02-21T23:41:03.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:03 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/3216798985' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-21T23:41:05.029 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:04 smithi090 ceph-mon[43385]: mgrmap e4: x(active, since 3s) 2024-02-21T23:41:05.029 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:04 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/1526553183' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-21T23:41:05.029 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:04 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: ignoring --setuser ceph since I am not root 2024-02-21T23:41:05.030 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:04 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: ignoring --setgroup ceph since I am not root 2024-02-21T23:41:05.099 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.123+0000 7fc87984b700 1 Processor -- start 2024-02-21T23:41:05.099 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.123+0000 7fc87984b700 1 -- start start 2024-02-21T23:41:05.099 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.123+0000 7fc87984b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc87407a2f0 0x7fc87407a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:05.099 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.123+0000 7fc87984b700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc87407c020 con 0x7fc87407a2f0 2024-02-21T23:41:05.099 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.123+0000 7fc872ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc87407a2f0 0x7fc87407a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:05.099 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.123+0000 7fc872ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc87407a2f0 0x7fc87407a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60082/0 (socket says 172.21.15.90:60082) 2024-02-21T23:41:05.099 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.123+0000 7fc872ffd700 1 -- 172.21.15.90:0/3982012001 learned_addr learned my addr 172.21.15.90:0/3982012001 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:05.099 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.123+0000 7fc872ffd700 1 -- 172.21.15.90:0/3982012001 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc87407ac50 con 0x7fc87407a2f0 2024-02-21T23:41:05.099 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.124+0000 7fc872ffd700 1 --2- 172.21.15.90:0/3982012001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc87407a2f0 0x7fc87407a710 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc8640098a0 tx=0x7fc864009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d588f158cd3672e server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:05.100 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.124+0000 7fc871ffb700 1 -- 172.21.15.90:0/3982012001 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc864004160 con 0x7fc87407a2f0 2024-02-21T23:41:05.100 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.124+0000 7fc871ffb700 1 -- 172.21.15.90:0/3982012001 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc8640042c0 con 0x7fc87407a2f0 2024-02-21T23:41:05.100 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.124+0000 7fc871ffb700 1 -- 172.21.15.90:0/3982012001 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc864011a00 con 0x7fc87407a2f0 2024-02-21T23:41:05.100 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.124+0000 7fc87984b700 1 -- 172.21.15.90:0/3982012001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc87407a2f0 msgr2=0x7fc87407a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:05.100 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.124+0000 7fc87984b700 1 --2- 172.21.15.90:0/3982012001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc87407a2f0 0x7fc87407a710 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc8640098a0 tx=0x7fc864009bb0 comp rx=0 tx=0).stop 2024-02-21T23:41:05.100 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.124+0000 7fc87984b700 1 -- 172.21.15.90:0/3982012001 shutdown_connections 2024-02-21T23:41:05.100 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.124+0000 7fc87984b700 1 --2- 172.21.15.90:0/3982012001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc87407a2f0 0x7fc87407a710 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:05.100 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.124+0000 7fc87984b700 1 -- 172.21.15.90:0/3982012001 >> 172.21.15.90:0/3982012001 conn(0x7fc874101a60 msgr2=0x7fc874103e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:05.100 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.124+0000 7fc87984b700 1 -- 172.21.15.90:0/3982012001 shutdown_connections 2024-02-21T23:41:05.100 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.124+0000 7fc87984b700 1 -- 172.21.15.90:0/3982012001 wait complete. 2024-02-21T23:41:05.100 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.125+0000 7fc87984b700 1 Processor -- start 2024-02-21T23:41:05.100 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.125+0000 7fc87984b700 1 -- start start 2024-02-21T23:41:05.100 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.125+0000 7fc87984b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc87407a2f0 0x7fc87419c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:05.100 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.125+0000 7fc87984b700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc87419c9e0 con 0x7fc87407a2f0 2024-02-21T23:41:05.101 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.125+0000 7fc872ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc87407a2f0 0x7fc87419c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:05.101 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.125+0000 7fc872ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc87407a2f0 0x7fc87419c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60090/0 (socket says 172.21.15.90:60090) 2024-02-21T23:41:05.101 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.125+0000 7fc872ffd700 1 -- 172.21.15.90:0/1526553183 learned_addr learned my addr 172.21.15.90:0/1526553183 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:05.101 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.126+0000 7fc872ffd700 1 -- 172.21.15.90:0/1526553183 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc864009550 con 0x7fc87407a2f0 2024-02-21T23:41:05.101 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.126+0000 7fc872ffd700 1 --2- 172.21.15.90:0/1526553183 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc87407a2f0 0x7fc87419c4a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc864009130 tx=0x7fc864011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:05.101 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.126+0000 7fc878849700 1 -- 172.21.15.90:0/1526553183 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc8640038b0 con 0x7fc87407a2f0 2024-02-21T23:41:05.101 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.126+0000 7fc878849700 1 -- 172.21.15.90:0/1526553183 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc864003a10 con 0x7fc87407a2f0 2024-02-21T23:41:05.101 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.126+0000 7fc878849700 1 -- 172.21.15.90:0/1526553183 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc864019550 con 0x7fc87407a2f0 2024-02-21T23:41:05.101 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.126+0000 7fc87984b700 1 -- 172.21.15.90:0/1526553183 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc87419cbe0 con 0x7fc87407a2f0 2024-02-21T23:41:05.101 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.126+0000 7fc87984b700 1 -- 172.21.15.90:0/1526553183 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc87419d080 con 0x7fc87407a2f0 2024-02-21T23:41:05.101 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.127+0000 7fc878849700 1 -- 172.21.15.90:0/1526553183 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 4) v1 ==== 45526+0+0 (secure 0 0 0) 0x7fc8640197c0 con 0x7fc87407a2f0 2024-02-21T23:41:05.101 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.127+0000 7fc87984b700 1 -- 172.21.15.90:0/1526553183 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc87410c930 con 0x7fc87407a2f0 2024-02-21T23:41:05.101 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.127+0000 7fc878849700 1 --2- 172.21.15.90:0/1526553183 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7fc85c038690 0x7fc85c03ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:05.101 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.127+0000 7fc878849700 1 -- 172.21.15.90:0/1526553183 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc86404c600 con 0x7fc87407a2f0 2024-02-21T23:41:05.102 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.127+0000 7fc8727fc700 1 --2- 172.21.15.90:0/1526553183 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7fc85c038690 0x7fc85c03ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:05.102 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.127+0000 7fc8727fc700 1 --2- 172.21.15.90:0/1526553183 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7fc85c038690 0x7fc85c03ab50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc868006fd0 tx=0x7fc868006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:05.102 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.132+0000 7fc878849700 1 -- 172.21.15.90:0/1526553183 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fc864018d20 con 0x7fc87407a2f0 2024-02-21T23:41:05.102 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.374+0000 7fc87984b700 1 -- 172.21.15.90:0/1526553183 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fc87410cb50 con 0x7fc87407a2f0 2024-02-21T23:41:05.102 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.741+0000 7fc878849700 1 -- 172.21.15.90:0/1526553183 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mgrmap(e 5) v1 ==== 45537+0+0 (secure 0 0 0) 0x7fc86402d960 con 0x7fc87407a2f0 2024-02-21T23:41:05.102 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.743+0000 7fc878849700 1 -- 172.21.15.90:0/1526553183 <== mon.0 v2:172.21.15.90:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fc86404e4d0 con 0x7fc87407a2f0 2024-02-21T23:41:05.102 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.745+0000 7fc87984b700 1 -- 172.21.15.90:0/1526553183 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7fc85c038690 msgr2=0x7fc85c03ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:05.102 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.745+0000 7fc87984b700 1 --2- 172.21.15.90:0/1526553183 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7fc85c038690 0x7fc85c03ab50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc868006fd0 tx=0x7fc868006e40 comp rx=0 tx=0).stop 2024-02-21T23:41:05.102 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.745+0000 7fc87984b700 1 -- 172.21.15.90:0/1526553183 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc87407a2f0 msgr2=0x7fc87419c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:05.102 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.745+0000 7fc87984b700 1 --2- 172.21.15.90:0/1526553183 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc87407a2f0 0x7fc87419c4a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc864009130 tx=0x7fc864011ad0 comp rx=0 tx=0).stop 2024-02-21T23:41:05.102 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.745+0000 7fc87984b700 1 -- 172.21.15.90:0/1526553183 shutdown_connections 2024-02-21T23:41:05.102 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.745+0000 7fc87984b700 1 --2- 172.21.15.90:0/1526553183 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7fc85c038690 0x7fc85c03ab50 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:05.102 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.745+0000 7fc87984b700 1 --2- 172.21.15.90:0/1526553183 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc87407a2f0 0x7fc87419c4a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:05.102 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.745+0000 7fc87984b700 1 -- 172.21.15.90:0/1526553183 >> 172.21.15.90:0/1526553183 conn(0x7fc874101a60 msgr2=0x7fc8741025d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:05.103 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.746+0000 7fc87984b700 1 -- 172.21.15.90:0/1526553183 shutdown_connections 2024-02-21T23:41:05.103 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:04.746+0000 7fc87984b700 1 -- 172.21.15.90:0/1526553183 wait complete. 2024-02-21T23:41:05.340 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:05 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:05.178+0000 7f6dd2408200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-21T23:41:05.340 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:05 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:05.268+0000 7f6dd2408200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-21T23:41:05.340 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:05 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:05.355+0000 7f6dd2408200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-21T23:41:05.727 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:05 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:05.446+0000 7f6dd2408200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-21T23:41:06.075 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:05 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/1526553183' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-21T23:41:06.076 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:05 smithi090 ceph-mon[43385]: mgrmap e5: x(active, since 4s) 2024-02-21T23:41:06.276 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout { 2024-02-21T23:41:06.276 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-02-21T23:41:06.276 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-21T23:41:06.276 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-02-21T23:41:06.276 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-21T23:41:06.276 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout } 2024-02-21T23:41:06.276 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.615+0000 7f7f5109c700 1 Processor -- start 2024-02-21T23:41:06.276 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.615+0000 7f7f5109c700 1 -- start start 2024-02-21T23:41:06.276 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.615+0000 7f7f5109c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f4c106f00 0x7f7f4c1092f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:06.277 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.615+0000 7f7f5109c700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4c109830 con 0x7f7f4c106f00 2024-02-21T23:41:06.277 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.615+0000 7f7f4ad9d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f4c106f00 0x7f7f4c1092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:06.277 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.615+0000 7f7f4ad9d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f4c106f00 0x7f7f4c1092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60116/0 (socket says 172.21.15.90:60116) 2024-02-21T23:41:06.277 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.615+0000 7f7f4ad9d700 1 -- 172.21.15.90:0/2191210396 learned_addr learned my addr 172.21.15.90:0/2191210396 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:06.277 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.616+0000 7f7f4ad9d700 1 -- 172.21.15.90:0/2191210396 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f4c109970 con 0x7f7f4c106f00 2024-02-21T23:41:06.277 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.616+0000 7f7f4ad9d700 1 --2- 172.21.15.90:0/2191210396 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f4c106f00 0x7f7f4c1092f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c0098a0 tx=0x7f7f3c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d775629d597287d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:06.277 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.616+0000 7f7f49d9b700 1 -- 172.21.15.90:0/2191210396 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7f3c004160 con 0x7f7f4c106f00 2024-02-21T23:41:06.277 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.616+0000 7f7f49d9b700 1 -- 172.21.15.90:0/2191210396 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f3c0042c0 con 0x7f7f4c106f00 2024-02-21T23:41:06.277 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.616+0000 7f7f49d9b700 1 -- 172.21.15.90:0/2191210396 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7f3c011a00 con 0x7f7f4c106f00 2024-02-21T23:41:06.277 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.616+0000 7f7f5109c700 1 -- 172.21.15.90:0/2191210396 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f4c106f00 msgr2=0x7f7f4c1092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:06.277 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.616+0000 7f7f5109c700 1 --2- 172.21.15.90:0/2191210396 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f4c106f00 0x7f7f4c1092f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c0098a0 tx=0x7f7f3c009bb0 comp rx=0 tx=0).stop 2024-02-21T23:41:06.278 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.617+0000 7f7f5109c700 1 -- 172.21.15.90:0/2191210396 shutdown_connections 2024-02-21T23:41:06.278 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.617+0000 7f7f5109c700 1 --2- 172.21.15.90:0/2191210396 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f4c106f00 0x7f7f4c1092f0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:06.278 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.617+0000 7f7f5109c700 1 -- 172.21.15.90:0/2191210396 >> 172.21.15.90:0/2191210396 conn(0x7f7f4c100970 msgr2=0x7f7f4c102d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:06.278 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.617+0000 7f7f5109c700 1 -- 172.21.15.90:0/2191210396 shutdown_connections 2024-02-21T23:41:06.278 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.617+0000 7f7f5109c700 1 -- 172.21.15.90:0/2191210396 wait complete. 2024-02-21T23:41:06.278 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.617+0000 7f7f5109c700 1 Processor -- start 2024-02-21T23:41:06.278 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.618+0000 7f7f5109c700 1 -- start start 2024-02-21T23:41:06.278 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.618+0000 7f7f5109c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f4c106f00 0x7f7f4c19a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:06.278 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.618+0000 7f7f5109c700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4c19a860 con 0x7f7f4c106f00 2024-02-21T23:41:06.278 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.618+0000 7f7f4ad9d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f4c106f00 0x7f7f4c19a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:06.278 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.618+0000 7f7f4ad9d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f4c106f00 0x7f7f4c19a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60128/0 (socket says 172.21.15.90:60128) 2024-02-21T23:41:06.278 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.618+0000 7f7f4ad9d700 1 -- 172.21.15.90:0/4150240897 learned_addr learned my addr 172.21.15.90:0/4150240897 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:06.278 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.618+0000 7f7f4ad9d700 1 -- 172.21.15.90:0/4150240897 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f3c009550 con 0x7f7f4c106f00 2024-02-21T23:41:06.279 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.618+0000 7f7f4ad9d700 1 --2- 172.21.15.90:0/4150240897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f4c106f00 0x7f7f4c19a320 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c011a20 tx=0x7f7f3c011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:06.279 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.618+0000 7f7f33fff700 1 -- 172.21.15.90:0/4150240897 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7f3c003af0 con 0x7f7f4c106f00 2024-02-21T23:41:06.279 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.618+0000 7f7f5109c700 1 -- 172.21.15.90:0/4150240897 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f4c19aa60 con 0x7f7f4c106f00 2024-02-21T23:41:06.279 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.618+0000 7f7f33fff700 1 -- 172.21.15.90:0/4150240897 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f3c003c50 con 0x7f7f4c106f00 2024-02-21T23:41:06.279 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.618+0000 7f7f33fff700 1 -- 172.21.15.90:0/4150240897 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7f3c019760 con 0x7f7f4c106f00 2024-02-21T23:41:06.279 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.619+0000 7f7f5109c700 1 -- 172.21.15.90:0/4150240897 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f4c19af00 con 0x7f7f4c106f00 2024-02-21T23:41:06.279 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.619+0000 7f7f33fff700 1 -- 172.21.15.90:0/4150240897 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 45537+0+0 (secure 0 0 0) 0x7f7f3c0199a0 con 0x7f7f4c106f00 2024-02-21T23:41:06.279 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.619+0000 7f7f33fff700 1 --2- 172.21.15.90:0/4150240897 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7f7f34038700 0x7f7f3403abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:06.279 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.619+0000 7f7f5109c700 1 -- 172.21.15.90:0/4150240897 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f4c10a710 con 0x7f7f4c106f00 2024-02-21T23:41:06.279 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.620+0000 7f7f4a59c700 1 -- 172.21.15.90:0/4150240897 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7f7f34038700 msgr2=0x7f7f3403abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/4247695574 2024-02-21T23:41:06.279 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.620+0000 7f7f33fff700 1 -- 172.21.15.90:0/4150240897 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7f3c04c930 con 0x7f7f4c106f00 2024-02-21T23:41:06.279 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.620+0000 7f7f4a59c700 1 --2- 172.21.15.90:0/4150240897 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7f7f34038700 0x7f7f3403abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-21T23:41:06.279 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.628+0000 7f7f33fff700 1 -- 172.21.15.90:0/4150240897 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f7f3c018db0 con 0x7f7f4c106f00 2024-02-21T23:41:06.279 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.821+0000 7f7f4a59c700 1 -- 172.21.15.90:0/4150240897 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7f7f34038700 msgr2=0x7f7f3403abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/4247695574 2024-02-21T23:41:06.280 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.821+0000 7f7f4a59c700 1 --2- 172.21.15.90:0/4150240897 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7f7f34038700 0x7f7f3403abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-21T23:41:06.280 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.917+0000 7f7f5109c700 1 -- 172.21.15.90:0/4150240897 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f7f4c19de40 con 0x7f7f4c106f00 2024-02-21T23:41:06.280 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.918+0000 7f7f33fff700 1 -- 172.21.15.90:0/4150240897 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f7f3c018350 con 0x7f7f4c106f00 2024-02-21T23:41:06.280 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.920+0000 7f7f5109c700 1 -- 172.21.15.90:0/4150240897 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7f7f34038700 msgr2=0x7f7f3403abc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-21T23:41:06.280 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.920+0000 7f7f5109c700 1 --2- 172.21.15.90:0/4150240897 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7f7f34038700 0x7f7f3403abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:06.280 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.920+0000 7f7f5109c700 1 -- 172.21.15.90:0/4150240897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f4c106f00 msgr2=0x7f7f4c19a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:06.280 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.920+0000 7f7f5109c700 1 --2- 172.21.15.90:0/4150240897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f4c106f00 0x7f7f4c19a320 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c011a20 tx=0x7f7f3c011b00 comp rx=0 tx=0).stop 2024-02-21T23:41:06.280 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.920+0000 7f7f5109c700 1 -- 172.21.15.90:0/4150240897 shutdown_connections 2024-02-21T23:41:06.281 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.920+0000 7f7f5109c700 1 --2- 172.21.15.90:0/4150240897 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7f7f34038700 0x7f7f3403abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:06.281 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.920+0000 7f7f5109c700 1 --2- 172.21.15.90:0/4150240897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f4c106f00 0x7f7f4c19a320 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:06.281 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.920+0000 7f7f5109c700 1 -- 172.21.15.90:0/4150240897 >> 172.21.15.90:0/4150240897 conn(0x7f7f4c100970 msgr2=0x7f7f4c101540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:06.281 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.921+0000 7f7f5109c700 1 -- 172.21.15.90:0/4150240897 shutdown_connections 2024-02-21T23:41:06.281 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:05.921+0000 7f7f5109c700 1 -- 172.21.15.90:0/4150240897 wait complete. 2024-02-21T23:41:06.281 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for the mgr to restart... 2024-02-21T23:41:06.281 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for mgr epoch 5... 2024-02-21T23:41:06.341 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:06 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:06.095+0000 7f6dd2408200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-21T23:41:06.702 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:06 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:06.356+0000 7f6dd2408200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-21T23:41:06.702 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:06 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:06.448+0000 7f6dd2408200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-21T23:41:06.702 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:06 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:06.559+0000 7f6dd2408200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-21T23:41:06.702 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:06 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:06.717+0000 7f6dd2408200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-21T23:41:06.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:06 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/4150240897' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-21T23:41:06.978 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:06 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:06.849+0000 7f6dd2408200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-21T23:41:06.978 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:06 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:06.934+0000 7f6dd2408200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-21T23:41:07.666 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:07 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:07.298+0000 7f6dd2408200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-21T23:41:07.666 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:07 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:07.681+0000 7f6dd2408200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-21T23:41:07.950 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:07 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:07.872+0000 7f6dd2408200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-21T23:41:07.951 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:07 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:07.965+0000 7f6dd2408200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-21T23:41:08.977 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:08 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:08.717+0000 7f6dd2408200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-21T23:41:08.977 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:08 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:08.814+0000 7f6dd2408200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-21T23:41:08.978 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:08 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:08.960+0000 7f6dd2408200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-21T23:41:09.965 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:09 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:09.605+0000 7f6dd2408200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-21T23:41:09.965 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:09 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:09.786+0000 7f6dd2408200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-21T23:41:09.965 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:09 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:09.979+0000 7f6dd2408200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-21T23:41:11.174 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:10 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:10.906+0000 7f6dd2408200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-21T23:41:11.174 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:11 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:11.189+0000 7f6dd2408200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-21T23:41:11.454 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:11 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:11.277+0000 7f6dd2408200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-21T23:41:11.727 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:11 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:11.469+0000 7f6dd2408200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-21T23:41:11.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:11 smithi090 ceph-mon[43385]: Active manager daemon x restarted 2024-02-21T23:41:11.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:11 smithi090 ceph-mon[43385]: Activating manager daemon x 2024-02-21T23:41:11.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:11 smithi090 ceph-mon[43385]: osdmap e2: 0 total, 0 up, 0 in 2024-02-21T23:41:11.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:11 smithi090 ceph-mon[43385]: mgrmap e6: x(active, starting, since 0.00552432s) 2024-02-21T23:41:11.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:11 smithi090 ceph-mon[43385]: from='mgr.14118 172.21.15.90:0/2013192540' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-21T23:41:11.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:11 smithi090 ceph-mon[43385]: from='mgr.14118 172.21.15.90:0/2013192540' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-21T23:41:11.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:11 smithi090 ceph-mon[43385]: from='mgr.14118 172.21.15.90:0/2013192540' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-21T23:41:11.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:11 smithi090 ceph-mon[43385]: from='mgr.14118 172.21.15.90:0/2013192540' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-21T23:41:11.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:11 smithi090 ceph-mon[43385]: from='mgr.14118 172.21.15.90:0/2013192540' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-21T23:41:11.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:11 smithi090 ceph-mon[43385]: Manager daemon x is now available 2024-02-21T23:41:11.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:11 smithi090 ceph-mon[43385]: from='mgr.14118 172.21.15.90:0/2013192540' entity='mgr.x' 2024-02-21T23:41:11.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:11 smithi090 ceph-mon[43385]: from='mgr.14118 172.21.15.90:0/2013192540' entity='mgr.x' 2024-02-21T23:41:11.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:11 smithi090 ceph-mon[43385]: from='mgr.14118 172.21.15.90:0/2013192540' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:41:12.889 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout { 2024-02-21T23:41:12.889 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-02-21T23:41:12.889 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-21T23:41:12.889 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout } 2024-02-21T23:41:12.889 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.785+0000 7effdb646700 1 Processor -- start 2024-02-21T23:41:12.889 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.786+0000 7effdb646700 1 -- start start 2024-02-21T23:41:12.889 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.786+0000 7effdb646700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7effd4106f60 0x7effd4109350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:12.889 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.786+0000 7effdb646700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd4079f40 con 0x7effd4106f60 2024-02-21T23:41:12.890 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.786+0000 7effd93e2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7effd4106f60 0x7effd4109350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:12.890 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.786+0000 7effd93e2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7effd4106f60 0x7effd4109350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60134/0 (socket says 172.21.15.90:60134) 2024-02-21T23:41:12.890 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.786+0000 7effd93e2700 1 -- 172.21.15.90:0/1298093025 learned_addr learned my addr 172.21.15.90:0/1298093025 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:12.890 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.786+0000 7effd93e2700 1 -- 172.21.15.90:0/1298093025 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd4109890 con 0x7effd4106f60 2024-02-21T23:41:12.890 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.786+0000 7effd93e2700 1 --2- 172.21.15.90:0/1298093025 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7effd4106f60 0x7effd4109350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7effd00098a0 tx=0x7effd0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45ef17268ff6b2f6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:12.890 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.787+0000 7effcbfff700 1 -- 172.21.15.90:0/1298093025 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7effd0004160 con 0x7effd4106f60 2024-02-21T23:41:12.890 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.787+0000 7effcbfff700 1 -- 172.21.15.90:0/1298093025 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7effd00042c0 con 0x7effd4106f60 2024-02-21T23:41:12.890 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.787+0000 7effcbfff700 1 -- 172.21.15.90:0/1298093025 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7effd0011a00 con 0x7effd4106f60 2024-02-21T23:41:12.890 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.787+0000 7effdb646700 1 -- 172.21.15.90:0/1298093025 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7effd4106f60 msgr2=0x7effd4109350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:12.890 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.787+0000 7effdb646700 1 --2- 172.21.15.90:0/1298093025 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7effd4106f60 0x7effd4109350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7effd00098a0 tx=0x7effd0009bb0 comp rx=0 tx=0).stop 2024-02-21T23:41:12.890 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.787+0000 7effdb646700 1 -- 172.21.15.90:0/1298093025 shutdown_connections 2024-02-21T23:41:12.890 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.787+0000 7effdb646700 1 --2- 172.21.15.90:0/1298093025 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7effd4106f60 0x7effd4109350 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:12.890 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.787+0000 7effdb646700 1 -- 172.21.15.90:0/1298093025 >> 172.21.15.90:0/1298093025 conn(0x7effd4100970 msgr2=0x7effd4102d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:12.890 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.788+0000 7effdb646700 1 -- 172.21.15.90:0/1298093025 shutdown_connections 2024-02-21T23:41:12.891 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.788+0000 7effdb646700 1 -- 172.21.15.90:0/1298093025 wait complete. 2024-02-21T23:41:12.891 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.788+0000 7effdb646700 1 Processor -- start 2024-02-21T23:41:12.891 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.788+0000 7effdb646700 1 -- start start 2024-02-21T23:41:12.891 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.789+0000 7effdb646700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7effd4106f60 0x7effd419a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:12.891 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.789+0000 7effdb646700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd419a850 con 0x7effd4106f60 2024-02-21T23:41:12.891 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.789+0000 7effd93e2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7effd4106f60 0x7effd419a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:12.891 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.789+0000 7effd93e2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7effd4106f60 0x7effd419a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60150/0 (socket says 172.21.15.90:60150) 2024-02-21T23:41:12.891 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.789+0000 7effd93e2700 1 -- 172.21.15.90:0/1751865229 learned_addr learned my addr 172.21.15.90:0/1751865229 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:12.891 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.789+0000 7effd93e2700 1 -- 172.21.15.90:0/1751865229 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd0009550 con 0x7effd4106f60 2024-02-21T23:41:12.891 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.789+0000 7effd93e2700 1 --2- 172.21.15.90:0/1751865229 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7effd4106f60 0x7effd419a310 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7effd0009130 tx=0x7effd0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:12.891 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.789+0000 7effca7fc700 1 -- 172.21.15.90:0/1751865229 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7effd00038b0 con 0x7effd4106f60 2024-02-21T23:41:12.891 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.789+0000 7effca7fc700 1 -- 172.21.15.90:0/1751865229 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7effd0003a10 con 0x7effd4106f60 2024-02-21T23:41:12.891 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.789+0000 7effdb646700 1 -- 172.21.15.90:0/1751865229 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effd419aa50 con 0x7effd4106f60 2024-02-21T23:41:12.891 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.789+0000 7effca7fc700 1 -- 172.21.15.90:0/1751865229 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7effd0019550 con 0x7effd4106f60 2024-02-21T23:41:12.892 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.789+0000 7effdb646700 1 -- 172.21.15.90:0/1751865229 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effd419aef0 con 0x7effd4106f60 2024-02-21T23:41:12.892 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.790+0000 7effca7fc700 1 -- 172.21.15.90:0/1751865229 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 45537+0+0 (secure 0 0 0) 0x7effd001a960 con 0x7effd4106f60 2024-02-21T23:41:12.892 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.791+0000 7effca7fc700 1 --2- 172.21.15.90:0/1751865229 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7effc0038690 0x7effc003ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:12.892 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.791+0000 7effca7fc700 1 -- 172.21.15.90:0/1751865229 --> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7effc003b260 con 0x7effc0038690 2024-02-21T23:41:12.892 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.791+0000 7effca7fc700 1 -- 172.21.15.90:0/1751865229 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7effd004c470 con 0x7effd4106f60 2024-02-21T23:41:12.892 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.791+0000 7effd8be1700 1 -- 172.21.15.90:0/1751865229 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7effc0038690 msgr2=0x7effc003ab50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/4247695574 2024-02-21T23:41:12.892 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.791+0000 7effd8be1700 1 --2- 172.21.15.90:0/1751865229 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7effc0038690 0x7effc003ab50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-21T23:41:12.892 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.992+0000 7effd8be1700 1 -- 172.21.15.90:0/1751865229 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7effc0038690 msgr2=0x7effc003ab50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/4247695574 2024-02-21T23:41:12.892 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:06.992+0000 7effd8be1700 1 --2- 172.21.15.90:0/1751865229 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7effc0038690 0x7effc003ab50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-21T23:41:12.892 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:07.393+0000 7effd8be1700 1 -- 172.21.15.90:0/1751865229 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7effc0038690 msgr2=0x7effc003ab50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/4247695574 2024-02-21T23:41:12.893 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:07.393+0000 7effd8be1700 1 --2- 172.21.15.90:0/1751865229 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7effc0038690 0x7effc003ab50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-02-21T23:41:12.893 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:08.194+0000 7effd8be1700 1 -- 172.21.15.90:0/1751865229 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7effc0038690 msgr2=0x7effc003ab50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/4247695574 2024-02-21T23:41:12.893 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:08.194+0000 7effd8be1700 1 --2- 172.21.15.90:0/1751865229 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7effc0038690 0x7effc003ab50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-02-21T23:41:12.893 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:12 smithi090 ceph-mon[43385]: Found migration_current of "None". Setting to last migration. 2024-02-21T23:41:12.893 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:12 smithi090 ceph-mon[43385]: from='mgr.14118 172.21.15.90:0/2013192540' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:41:12.893 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:12 smithi090 ceph-mon[43385]: from='mgr.14118 172.21.15.90:0/2013192540' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:41:12.893 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:12 smithi090 ceph-mon[43385]: from='mgr.14118 172.21.15.90:0/2013192540' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-21T23:41:12.893 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:12 smithi090 ceph-mon[43385]: from='mgr.14118 172.21.15.90:0/2013192540' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-21T23:41:12.893 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:12 smithi090 ceph-mon[43385]: from='mgr.14118 172.21.15.90:0/2013192540' entity='mgr.x' 2024-02-21T23:41:12.893 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:12 smithi090 ceph-mon[43385]: from='mgr.14118 172.21.15.90:0/2013192540' entity='mgr.x' 2024-02-21T23:41:12.894 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:12 smithi090 ceph-mon[43385]: from='mgr.14118 172.21.15.90:0/2013192540' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:41:12.894 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:12 smithi090 ceph-mon[43385]: mgrmap e7: x(active, since 1.00539s) 2024-02-21T23:41:12.896 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:09.795+0000 7effd8be1700 1 -- 172.21.15.90:0/1751865229 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7effc0038690 msgr2=0x7effc003ab50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/4247695574 2024-02-21T23:41:12.896 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:09.795+0000 7effd8be1700 1 --2- 172.21.15.90:0/1751865229 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7effc0038690 0x7effc003ab50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-02-21T23:41:12.896 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:11.487+0000 7effca7fc700 1 -- 172.21.15.90:0/1751865229 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mgrmap(e 6) v1 ==== 45304+0+0 (secure 0 0 0) 0x7effd0019d30 con 0x7effd4106f60 2024-02-21T23:41:12.896 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:11.487+0000 7effca7fc700 1 -- 172.21.15.90:0/1751865229 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7effc0038690 msgr2=0x7effc003ab50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-21T23:41:12.896 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:11.487+0000 7effca7fc700 1 --2- 172.21.15.90:0/1751865229 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7effc0038690 0x7effc003ab50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:12.896 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:12.486+0000 7effca7fc700 1 -- 172.21.15.90:0/1751865229 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mgrmap(e 7) v1 ==== 45431+0+0 (secure 0 0 0) 0x7effd004e050 con 0x7effd4106f60 2024-02-21T23:41:12.896 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:12.486+0000 7effca7fc700 1 --2- 172.21.15.90:0/1751865229 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7effc003bf70 0x7effc003e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:12.896 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:12.486+0000 7effca7fc700 1 -- 172.21.15.90:0/1751865229 --> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7effc003b260 con 0x7effc003bf70 2024-02-21T23:41:12.896 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:12.486+0000 7effd8be1700 1 --2- 172.21.15.90:0/1751865229 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7effc003bf70 0x7effc003e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:12.896 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:12.487+0000 7effd8be1700 1 --2- 172.21.15.90:0/1751865229 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7effc003bf70 0x7effc003e360 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7effc4003a10 tx=0x7effc4009290 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:12.896 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:12.487+0000 7effca7fc700 1 -- 172.21.15.90:0/1751865229 <== mgr.14118 v2:172.21.15.90:6800/1385551604 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7effc003b260 con 0x7effc003bf70 2024-02-21T23:41:12.896 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:12.493+0000 7effdb646700 1 -- 172.21.15.90:0/1751865229 --> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7effd40623a0 con 0x7effc003bf70 2024-02-21T23:41:12.897 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:12.494+0000 7effca7fc700 1 -- 172.21.15.90:0/1751865229 <== mgr.14118 v2:172.21.15.90:6800/1385551604 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7effd40623a0 con 0x7effc003bf70 2024-02-21T23:41:12.897 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:12.494+0000 7effdb646700 1 -- 172.21.15.90:0/1751865229 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7effc003bf70 msgr2=0x7effc003e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:12.897 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:12.494+0000 7effdb646700 1 --2- 172.21.15.90:0/1751865229 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7effc003bf70 0x7effc003e360 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7effc4003a10 tx=0x7effc4009290 comp rx=0 tx=0).stop 2024-02-21T23:41:12.897 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:12.494+0000 7effdb646700 1 -- 172.21.15.90:0/1751865229 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7effd4106f60 msgr2=0x7effd419a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:12.897 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:12.494+0000 7effdb646700 1 --2- 172.21.15.90:0/1751865229 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7effd4106f60 0x7effd419a310 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7effd0009130 tx=0x7effd0011ad0 comp rx=0 tx=0).stop 2024-02-21T23:41:12.897 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:12.494+0000 7effdb646700 1 -- 172.21.15.90:0/1751865229 shutdown_connections 2024-02-21T23:41:12.897 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:12.494+0000 7effdb646700 1 --2- 172.21.15.90:0/1751865229 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7effd4106f60 0x7effd419a310 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:12.897 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:12.494+0000 7effdb646700 1 --2- 172.21.15.90:0/1751865229 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7effc003bf70 0x7effc003e360 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:12.898 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:12.494+0000 7effdb646700 1 --2- 172.21.15.90:0/1751865229 >> [v2:172.21.15.90:6800/4247695574,v1:172.21.15.90:6801/4247695574] conn(0x7effc0038690 0x7effc003ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:12.898 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:12.494+0000 7effdb646700 1 -- 172.21.15.90:0/1751865229 >> 172.21.15.90:0/1751865229 conn(0x7effd4100970 msgr2=0x7effd4101540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:12.898 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:12.494+0000 7effdb646700 1 -- 172.21.15.90:0/1751865229 shutdown_connections 2024-02-21T23:41:12.898 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:12.494+0000 7effdb646700 1 -- 172.21.15.90:0/1751865229 wait complete. 2024-02-21T23:41:12.898 INFO:teuthology.orchestra.run.smithi090.stdout:mgr epoch 5 is available 2024-02-21T23:41:12.898 INFO:teuthology.orchestra.run.smithi090.stdout:Setting orchestrator backend to cephadm... 2024-02-21T23:41:13.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:13 smithi090 ceph-mon[43385]: [21/Feb/2024:23:41:12] ENGINE Bus STARTING 2024-02-21T23:41:13.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:13 smithi090 ceph-mon[43385]: [21/Feb/2024:23:41:12] ENGINE Serving on http://172.21.15.90:8765 2024-02-21T23:41:13.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:13 smithi090 ceph-mon[43385]: [21/Feb/2024:23:41:12] ENGINE Serving on https://172.21.15.90:7150 2024-02-21T23:41:13.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:13 smithi090 ceph-mon[43385]: [21/Feb/2024:23:41:12] ENGINE Bus STARTED 2024-02-21T23:41:13.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:13 smithi090 ceph-mon[43385]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-21T23:41:13.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:13 smithi090 ceph-mon[43385]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-21T23:41:13.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:13 smithi090 ceph-mon[43385]: from='mgr.14118 172.21.15.90:0/2013192540' entity='mgr.x' 2024-02-21T23:41:13.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:13 smithi090 ceph-mon[43385]: from='mgr.14118 172.21.15.90:0/2013192540' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:41:13.848 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.224+0000 7f4d230b0700 1 Processor -- start 2024-02-21T23:41:13.848 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.224+0000 7f4d230b0700 1 -- start start 2024-02-21T23:41:13.848 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.224+0000 7f4d230b0700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d1c104d60 0x7f4d1c105180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:13.848 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.224+0000 7f4d230b0700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1c079f40 con 0x7f4d1c104d60 2024-02-21T23:41:13.848 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.224+0000 7f4d20e4c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d1c104d60 0x7f4d1c105180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:13.848 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.224+0000 7f4d20e4c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d1c104d60 0x7f4d1c105180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40978/0 (socket says 172.21.15.90:40978) 2024-02-21T23:41:13.849 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.224+0000 7f4d20e4c700 1 -- 172.21.15.90:0/4016838756 learned_addr learned my addr 172.21.15.90:0/4016838756 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:13.849 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.224+0000 7f4d20e4c700 1 -- 172.21.15.90:0/4016838756 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d1c1056c0 con 0x7f4d1c104d60 2024-02-21T23:41:13.849 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.225+0000 7f4d20e4c700 1 --2- 172.21.15.90:0/4016838756 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d1c104d60 0x7f4d1c105180 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4d100098a0 tx=0x7f4d10009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f91e6dbe14f7c497 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:13.849 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.225+0000 7f4d1b7fe700 1 -- 172.21.15.90:0/4016838756 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d10004160 con 0x7f4d1c104d60 2024-02-21T23:41:13.849 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.225+0000 7f4d1b7fe700 1 -- 172.21.15.90:0/4016838756 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4d100042c0 con 0x7f4d1c104d60 2024-02-21T23:41:13.849 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.225+0000 7f4d1b7fe700 1 -- 172.21.15.90:0/4016838756 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d10011a00 con 0x7f4d1c104d60 2024-02-21T23:41:13.849 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.225+0000 7f4d230b0700 1 -- 172.21.15.90:0/4016838756 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d1c104d60 msgr2=0x7f4d1c105180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:13.849 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.225+0000 7f4d230b0700 1 --2- 172.21.15.90:0/4016838756 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d1c104d60 0x7f4d1c105180 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4d100098a0 tx=0x7f4d10009bb0 comp rx=0 tx=0).stop 2024-02-21T23:41:13.849 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.225+0000 7f4d230b0700 1 -- 172.21.15.90:0/4016838756 shutdown_connections 2024-02-21T23:41:13.849 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.225+0000 7f4d230b0700 1 --2- 172.21.15.90:0/4016838756 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d1c104d60 0x7f4d1c105180 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:13.849 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.225+0000 7f4d230b0700 1 -- 172.21.15.90:0/4016838756 >> 172.21.15.90:0/4016838756 conn(0x7f4d1c1009a0 msgr2=0x7f4d1c102dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:13.849 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.225+0000 7f4d230b0700 1 -- 172.21.15.90:0/4016838756 shutdown_connections 2024-02-21T23:41:13.850 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.225+0000 7f4d230b0700 1 -- 172.21.15.90:0/4016838756 wait complete. 2024-02-21T23:41:13.850 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.226+0000 7f4d230b0700 1 Processor -- start 2024-02-21T23:41:13.850 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.226+0000 7f4d230b0700 1 -- start start 2024-02-21T23:41:13.850 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.226+0000 7f4d230b0700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d1c104d60 0x7f4d1c19c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:13.850 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.226+0000 7f4d230b0700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1c19ca90 con 0x7f4d1c104d60 2024-02-21T23:41:13.850 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.226+0000 7f4d20e4c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d1c104d60 0x7f4d1c19c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:13.850 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.226+0000 7f4d20e4c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d1c104d60 0x7f4d1c19c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40982/0 (socket says 172.21.15.90:40982) 2024-02-21T23:41:13.850 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.226+0000 7f4d20e4c700 1 -- 172.21.15.90:0/3081389524 learned_addr learned my addr 172.21.15.90:0/3081389524 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:13.850 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.227+0000 7f4d20e4c700 1 -- 172.21.15.90:0/3081389524 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d10009550 con 0x7f4d1c104d60 2024-02-21T23:41:13.850 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.227+0000 7f4d20e4c700 1 --2- 172.21.15.90:0/3081389524 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d1c104d60 0x7f4d1c19c550 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4d10009130 tx=0x7f4d10011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:13.850 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.227+0000 7f4d19ffb700 1 -- 172.21.15.90:0/3081389524 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d100038b0 con 0x7f4d1c104d60 2024-02-21T23:41:13.850 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.227+0000 7f4d19ffb700 1 -- 172.21.15.90:0/3081389524 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4d10003a10 con 0x7f4d1c104d60 2024-02-21T23:41:13.851 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.227+0000 7f4d230b0700 1 -- 172.21.15.90:0/3081389524 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d1c19cc90 con 0x7f4d1c104d60 2024-02-21T23:41:13.851 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.227+0000 7f4d19ffb700 1 -- 172.21.15.90:0/3081389524 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d10019550 con 0x7f4d1c104d60 2024-02-21T23:41:13.851 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.227+0000 7f4d230b0700 1 -- 172.21.15.90:0/3081389524 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d1c19d130 con 0x7f4d1c104d60 2024-02-21T23:41:13.851 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.228+0000 7f4d19ffb700 1 -- 172.21.15.90:0/3081389524 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 7) v1 ==== 45431+0+0 (secure 0 0 0) 0x7f4d10019810 con 0x7f4d1c104d60 2024-02-21T23:41:13.851 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.228+0000 7f4d230b0700 1 -- 172.21.15.90:0/3081389524 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d1c196220 con 0x7f4d1c104d60 2024-02-21T23:41:13.851 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.228+0000 7f4d19ffb700 1 --2- 172.21.15.90:0/3081389524 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f4d0c038570 0x7f4d0c03aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:13.851 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.228+0000 7f4d19ffb700 1 -- 172.21.15.90:0/3081389524 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f4d1004c650 con 0x7f4d1c104d60 2024-02-21T23:41:13.851 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.228+0000 7f4d1bfff700 1 --2- 172.21.15.90:0/3081389524 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f4d0c038570 0x7f4d0c03aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:13.851 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.228+0000 7f4d1bfff700 1 --2- 172.21.15.90:0/3081389524 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f4d0c038570 0x7f4d0c03aa30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4d08006fd0 tx=0x7f4d08006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:13.851 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.233+0000 7f4d19ffb700 1 -- 172.21.15.90:0/3081389524 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f4d1002b340 con 0x7f4d1c104d60 2024-02-21T23:41:13.851 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.438+0000 7f4d230b0700 1 -- 172.21.15.90:0/3081389524 --> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f4d1c0611b0 con 0x7f4d0c038570 2024-02-21T23:41:13.851 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.446+0000 7f4d19ffb700 1 -- 172.21.15.90:0/3081389524 <== mgr.14118 v2:172.21.15.90:6800/1385551604 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4d1c0611b0 con 0x7f4d0c038570 2024-02-21T23:41:13.852 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.448+0000 7f4d230b0700 1 -- 172.21.15.90:0/3081389524 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f4d0c038570 msgr2=0x7f4d0c03aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:13.852 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.448+0000 7f4d230b0700 1 --2- 172.21.15.90:0/3081389524 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f4d0c038570 0x7f4d0c03aa30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4d08006fd0 tx=0x7f4d08006e40 comp rx=0 tx=0).stop 2024-02-21T23:41:13.852 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.448+0000 7f4d230b0700 1 -- 172.21.15.90:0/3081389524 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d1c104d60 msgr2=0x7f4d1c19c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:13.852 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.448+0000 7f4d230b0700 1 --2- 172.21.15.90:0/3081389524 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d1c104d60 0x7f4d1c19c550 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4d10009130 tx=0x7f4d10011ad0 comp rx=0 tx=0).stop 2024-02-21T23:41:13.852 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.448+0000 7f4d230b0700 1 -- 172.21.15.90:0/3081389524 shutdown_connections 2024-02-21T23:41:13.852 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.448+0000 7f4d230b0700 1 --2- 172.21.15.90:0/3081389524 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f4d0c038570 0x7f4d0c03aa30 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:13.852 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.448+0000 7f4d230b0700 1 --2- 172.21.15.90:0/3081389524 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d1c104d60 0x7f4d1c19c550 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:13.852 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.448+0000 7f4d230b0700 1 -- 172.21.15.90:0/3081389524 >> 172.21.15.90:0/3081389524 conn(0x7f4d1c1009a0 msgr2=0x7f4d1c193360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:13.852 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.448+0000 7f4d230b0700 1 -- 172.21.15.90:0/3081389524 shutdown_connections 2024-02-21T23:41:13.852 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:13.448+0000 7f4d230b0700 1 -- 172.21.15.90:0/3081389524 wait complete. 2024-02-21T23:41:14.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:14 smithi090 ceph-mon[43385]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:41:14.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:14 smithi090 ceph-mon[43385]: mgrmap e8: x(active, since 2s) 2024-02-21T23:41:14.776 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout value unchanged 2024-02-21T23:41:14.776 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.189+0000 7fd78bff5700 1 Processor -- start 2024-02-21T23:41:14.776 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.189+0000 7fd78bff5700 1 -- start start 2024-02-21T23:41:14.776 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.189+0000 7fd78bff5700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd784106f00 0x7fd7841092f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:14.776 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.189+0000 7fd78bff5700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd784109830 con 0x7fd784106f00 2024-02-21T23:41:14.776 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.189+0000 7fd789d91700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd784106f00 0x7fd7841092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:14.776 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.189+0000 7fd789d91700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd784106f00 0x7fd7841092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40994/0 (socket says 172.21.15.90:40994) 2024-02-21T23:41:14.776 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.189+0000 7fd789d91700 1 -- 172.21.15.90:0/1674448635 learned_addr learned my addr 172.21.15.90:0/1674448635 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:14.777 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.190+0000 7fd789d91700 1 -- 172.21.15.90:0/1674448635 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd784109970 con 0x7fd784106f00 2024-02-21T23:41:14.777 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.190+0000 7fd789d91700 1 --2- 172.21.15.90:0/1674448635 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd784106f00 0x7fd7841092f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd7800098a0 tx=0x7fd780009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=508a2d18fd062d70 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:14.777 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.190+0000 7fd788d8f700 1 -- 172.21.15.90:0/1674448635 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd780004160 con 0x7fd784106f00 2024-02-21T23:41:14.777 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.190+0000 7fd788d8f700 1 -- 172.21.15.90:0/1674448635 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd7800042c0 con 0x7fd784106f00 2024-02-21T23:41:14.777 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.190+0000 7fd788d8f700 1 -- 172.21.15.90:0/1674448635 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd780011a00 con 0x7fd784106f00 2024-02-21T23:41:14.777 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.190+0000 7fd78bff5700 1 -- 172.21.15.90:0/1674448635 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd784106f00 msgr2=0x7fd7841092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:14.777 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.190+0000 7fd78bff5700 1 --2- 172.21.15.90:0/1674448635 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd784106f00 0x7fd7841092f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd7800098a0 tx=0x7fd780009bb0 comp rx=0 tx=0).stop 2024-02-21T23:41:14.777 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.190+0000 7fd78bff5700 1 -- 172.21.15.90:0/1674448635 shutdown_connections 2024-02-21T23:41:14.777 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.190+0000 7fd78bff5700 1 --2- 172.21.15.90:0/1674448635 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd784106f00 0x7fd7841092f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:14.777 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.190+0000 7fd78bff5700 1 -- 172.21.15.90:0/1674448635 >> 172.21.15.90:0/1674448635 conn(0x7fd784100970 msgr2=0x7fd784102d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:14.777 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.191+0000 7fd78bff5700 1 -- 172.21.15.90:0/1674448635 shutdown_connections 2024-02-21T23:41:14.778 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.191+0000 7fd78bff5700 1 -- 172.21.15.90:0/1674448635 wait complete. 2024-02-21T23:41:14.778 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.191+0000 7fd78bff5700 1 Processor -- start 2024-02-21T23:41:14.778 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.191+0000 7fd78bff5700 1 -- start start 2024-02-21T23:41:14.778 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.192+0000 7fd78bff5700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd784106f00 0x7fd78419a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:14.778 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.192+0000 7fd78bff5700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd78419a860 con 0x7fd784106f00 2024-02-21T23:41:14.778 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.192+0000 7fd789d91700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd784106f00 0x7fd78419a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:14.778 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.192+0000 7fd789d91700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd784106f00 0x7fd78419a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40998/0 (socket says 172.21.15.90:40998) 2024-02-21T23:41:14.778 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.192+0000 7fd789d91700 1 -- 172.21.15.90:0/2177824926 learned_addr learned my addr 172.21.15.90:0/2177824926 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:14.778 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.192+0000 7fd789d91700 1 -- 172.21.15.90:0/2177824926 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd780009550 con 0x7fd784106f00 2024-02-21T23:41:14.778 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.192+0000 7fd789d91700 1 --2- 172.21.15.90:0/2177824926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd784106f00 0x7fd78419a320 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd780011a20 tx=0x7fd780011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:14.778 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.192+0000 7fd77affd700 1 -- 172.21.15.90:0/2177824926 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd780003af0 con 0x7fd784106f00 2024-02-21T23:41:14.779 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.192+0000 7fd77affd700 1 -- 172.21.15.90:0/2177824926 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd780003c50 con 0x7fd784106f00 2024-02-21T23:41:14.779 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.192+0000 7fd78bff5700 1 -- 172.21.15.90:0/2177824926 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd78419aa60 con 0x7fd784106f00 2024-02-21T23:41:14.779 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.192+0000 7fd77affd700 1 -- 172.21.15.90:0/2177824926 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd780019760 con 0x7fd784106f00 2024-02-21T23:41:14.779 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.192+0000 7fd78bff5700 1 -- 172.21.15.90:0/2177824926 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd78419af00 con 0x7fd784106f00 2024-02-21T23:41:14.779 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.193+0000 7fd77affd700 1 -- 172.21.15.90:0/2177824926 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 7) v1 ==== 45431+0+0 (secure 0 0 0) 0x7fd780019950 con 0x7fd784106f00 2024-02-21T23:41:14.779 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.193+0000 7fd78bff5700 1 -- 172.21.15.90:0/2177824926 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd78410a710 con 0x7fd784106f00 2024-02-21T23:41:14.779 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.193+0000 7fd77affd700 1 --2- 172.21.15.90:0/2177824926 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7fd770038620 0x7fd77003aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:14.779 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.193+0000 7fd77affd700 1 -- 172.21.15.90:0/2177824926 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fd78004c880 con 0x7fd784106f00 2024-02-21T23:41:14.779 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.193+0000 7fd789590700 1 --2- 172.21.15.90:0/2177824926 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7fd770038620 0x7fd77003aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:14.779 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.194+0000 7fd789590700 1 --2- 172.21.15.90:0/2177824926 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7fd770038620 0x7fd77003aae0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd774006fd0 tx=0x7fd774006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:14.780 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.198+0000 7fd77affd700 1 -- 172.21.15.90:0/2177824926 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fd780018530 con 0x7fd784106f00 2024-02-21T23:41:14.780 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.390+0000 7fd78bff5700 1 -- 172.21.15.90:0/2177824926 --> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fd784101c50 con 0x7fd770038620 2024-02-21T23:41:14.780 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.391+0000 7fd77affd700 1 -- 172.21.15.90:0/2177824926 <== mgr.14118 v2:172.21.15.90:6800/1385551604 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fd784101c50 con 0x7fd770038620 2024-02-21T23:41:14.780 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.393+0000 7fd78bff5700 1 -- 172.21.15.90:0/2177824926 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7fd770038620 msgr2=0x7fd77003aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:14.780 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.393+0000 7fd78bff5700 1 --2- 172.21.15.90:0/2177824926 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7fd770038620 0x7fd77003aae0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd774006fd0 tx=0x7fd774006e40 comp rx=0 tx=0).stop 2024-02-21T23:41:14.780 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.393+0000 7fd78bff5700 1 -- 172.21.15.90:0/2177824926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd784106f00 msgr2=0x7fd78419a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:14.780 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.393+0000 7fd78bff5700 1 --2- 172.21.15.90:0/2177824926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd784106f00 0x7fd78419a320 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd780011a20 tx=0x7fd780011b00 comp rx=0 tx=0).stop 2024-02-21T23:41:14.780 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.393+0000 7fd78bff5700 1 -- 172.21.15.90:0/2177824926 shutdown_connections 2024-02-21T23:41:14.780 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.393+0000 7fd78bff5700 1 --2- 172.21.15.90:0/2177824926 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7fd770038620 0x7fd77003aae0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:14.780 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.393+0000 7fd78bff5700 1 --2- 172.21.15.90:0/2177824926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd784106f00 0x7fd78419a320 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:14.780 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.393+0000 7fd78bff5700 1 -- 172.21.15.90:0/2177824926 >> 172.21.15.90:0/2177824926 conn(0x7fd784100970 msgr2=0x7fd784101540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:14.780 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.393+0000 7fd78bff5700 1 -- 172.21.15.90:0/2177824926 shutdown_connections 2024-02-21T23:41:14.781 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:14.393+0000 7fd78bff5700 1 -- 172.21.15.90:0/2177824926 wait complete. 2024-02-21T23:41:14.781 INFO:teuthology.orchestra.run.smithi090.stdout:Generating ssh key... 2024-02-21T23:41:15.795 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:15 smithi090 ceph-mon[43385]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:41:15.796 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:15 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: Generating public/private rsa key pair. 2024-02-21T23:41:15.796 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:15 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: Your identification has been saved in /tmp/tmpgnl9cblb/key. 2024-02-21T23:41:15.796 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:15 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: Your public key has been saved in /tmp/tmpgnl9cblb/key.pub. 2024-02-21T23:41:15.796 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:15 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: The key fingerprint is: 2024-02-21T23:41:15.796 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:15 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: SHA256:xmeSl+3tm4l1eqCKUjaZ2wiFBSsow6yuUwr+uCw+NTQ ceph-7ed81218-d112-11ee-95bd-87774f69a715 2024-02-21T23:41:15.796 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:15 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: The key's randomart image is: 2024-02-21T23:41:15.796 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:15 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: +---[RSA 3072]----+ 2024-02-21T23:41:15.796 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:15 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: | .. | 2024-02-21T23:41:15.796 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:15 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: |o . .. | 2024-02-21T23:41:15.796 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:15 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: |oo. . .o | 2024-02-21T23:41:15.797 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:15 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: |.o E ..... o | 2024-02-21T23:41:15.797 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:15 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: |. . . .So= . | 2024-02-21T23:41:15.797 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:15 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: |o .o ..*= . .. | 2024-02-21T23:41:15.797 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:15 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: |o+. . + = ..o..| 2024-02-21T23:41:15.797 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:15 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: |*o. . o.. .+ =.| 2024-02-21T23:41:15.797 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:15 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: |+*+. .. ... *o | 2024-02-21T23:41:15.797 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:15 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: +----[SHA256]-----+ 2024-02-21T23:41:16.134 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.121+0000 7fe6b7f3c700 1 Processor -- start 2024-02-21T23:41:16.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.121+0000 7fe6b7f3c700 1 -- start start 2024-02-21T23:41:16.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.121+0000 7fe6b7f3c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6b01059d0 0x7fe6b0105df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:16.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.121+0000 7fe6b7f3c700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b01044f0 con 0x7fe6b01059d0 2024-02-21T23:41:16.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.121+0000 7fe6b5cd8700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6b01059d0 0x7fe6b0105df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:16.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.121+0000 7fe6b5cd8700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6b01059d0 0x7fe6b0105df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41012/0 (socket says 172.21.15.90:41012) 2024-02-21T23:41:16.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.121+0000 7fe6b5cd8700 1 -- 172.21.15.90:0/845152119 learned_addr learned my addr 172.21.15.90:0/845152119 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:16.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.121+0000 7fe6b5cd8700 1 -- 172.21.15.90:0/845152119 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b0104630 con 0x7fe6b01059d0 2024-02-21T23:41:16.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.122+0000 7fe6b5cd8700 1 --2- 172.21.15.90:0/845152119 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6b01059d0 0x7fe6b0105df0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac0098a0 tx=0x7fe6ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8af29d93d0c66134 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:16.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.122+0000 7fe6b4cd6700 1 -- 172.21.15.90:0/845152119 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe6ac004160 con 0x7fe6b01059d0 2024-02-21T23:41:16.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.122+0000 7fe6b4cd6700 1 -- 172.21.15.90:0/845152119 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe6ac0042c0 con 0x7fe6b01059d0 2024-02-21T23:41:16.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.122+0000 7fe6b4cd6700 1 -- 172.21.15.90:0/845152119 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe6ac011a00 con 0x7fe6b01059d0 2024-02-21T23:41:16.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.122+0000 7fe6b7f3c700 1 -- 172.21.15.90:0/845152119 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6b01059d0 msgr2=0x7fe6b0105df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:16.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.122+0000 7fe6b7f3c700 1 --2- 172.21.15.90:0/845152119 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6b01059d0 0x7fe6b0105df0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac0098a0 tx=0x7fe6ac009bb0 comp rx=0 tx=0).stop 2024-02-21T23:41:16.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.122+0000 7fe6b7f3c700 1 -- 172.21.15.90:0/845152119 shutdown_connections 2024-02-21T23:41:16.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.122+0000 7fe6b7f3c700 1 --2- 172.21.15.90:0/845152119 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6b01059d0 0x7fe6b0105df0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:16.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.122+0000 7fe6b7f3c700 1 -- 172.21.15.90:0/845152119 >> 172.21.15.90:0/845152119 conn(0x7fe6b01000d0 msgr2=0x7fe6b0102510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:16.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.122+0000 7fe6b7f3c700 1 -- 172.21.15.90:0/845152119 shutdown_connections 2024-02-21T23:41:16.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.122+0000 7fe6b7f3c700 1 -- 172.21.15.90:0/845152119 wait complete. 2024-02-21T23:41:16.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.123+0000 7fe6b7f3c700 1 Processor -- start 2024-02-21T23:41:16.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.123+0000 7fe6b7f3c700 1 -- start start 2024-02-21T23:41:16.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.123+0000 7fe6b7f3c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6b01059d0 0x7fe6b019c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:16.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.123+0000 7fe6b7f3c700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b019ca70 con 0x7fe6b01059d0 2024-02-21T23:41:16.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.123+0000 7fe6b5cd8700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6b01059d0 0x7fe6b019c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:16.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.123+0000 7fe6b5cd8700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6b01059d0 0x7fe6b019c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41018/0 (socket says 172.21.15.90:41018) 2024-02-21T23:41:16.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.123+0000 7fe6b5cd8700 1 -- 172.21.15.90:0/2964305176 learned_addr learned my addr 172.21.15.90:0/2964305176 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:16.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.123+0000 7fe6b5cd8700 1 -- 172.21.15.90:0/2964305176 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6ac009550 con 0x7fe6b01059d0 2024-02-21T23:41:16.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.124+0000 7fe6b5cd8700 1 --2- 172.21.15.90:0/2964305176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6b01059d0 0x7fe6b019c530 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac009130 tx=0x7fe6ac011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:16.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.124+0000 7fe6a6ffd700 1 -- 172.21.15.90:0/2964305176 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe6ac0038b0 con 0x7fe6b01059d0 2024-02-21T23:41:16.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.124+0000 7fe6a6ffd700 1 -- 172.21.15.90:0/2964305176 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe6ac003a10 con 0x7fe6b01059d0 2024-02-21T23:41:16.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.124+0000 7fe6b7f3c700 1 -- 172.21.15.90:0/2964305176 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6b019cc70 con 0x7fe6b01059d0 2024-02-21T23:41:16.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.124+0000 7fe6a6ffd700 1 -- 172.21.15.90:0/2964305176 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe6ac019550 con 0x7fe6b01059d0 2024-02-21T23:41:16.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.124+0000 7fe6b7f3c700 1 -- 172.21.15.90:0/2964305176 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6b019d110 con 0x7fe6b01059d0 2024-02-21T23:41:16.142 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.124+0000 7fe6a6ffd700 1 -- 172.21.15.90:0/2964305176 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) v1 ==== 45537+0+0 (secure 0 0 0) 0x7fe6ac0197c0 con 0x7fe6b01059d0 2024-02-21T23:41:16.142 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.125+0000 7fe6a6ffd700 1 --2- 172.21.15.90:0/2964305176 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7fe69c0386e0 0x7fe69c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:16.142 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.125+0000 7fe6b7f3c700 1 -- 172.21.15.90:0/2964305176 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6b0196160 con 0x7fe6b01059d0 2024-02-21T23:41:16.142 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.125+0000 7fe6a6ffd700 1 -- 172.21.15.90:0/2964305176 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fe6ac04c760 con 0x7fe6b01059d0 2024-02-21T23:41:16.142 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.125+0000 7fe6b54d7700 1 --2- 172.21.15.90:0/2964305176 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7fe69c0386e0 0x7fe69c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:16.142 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.125+0000 7fe6b54d7700 1 --2- 172.21.15.90:0/2964305176 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7fe69c0386e0 0x7fe69c03aba0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0006fd0 tx=0x7fe6a0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:16.142 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.130+0000 7fe6a6ffd700 1 -- 172.21.15.90:0/2964305176 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fe6ac02b340 con 0x7fe6b01059d0 2024-02-21T23:41:16.142 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.322+0000 7fe6b7f3c700 1 -- 172.21.15.90:0/2964305176 --> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fe6b00611b0 con 0x7fe69c0386e0 2024-02-21T23:41:16.142 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.785+0000 7fe6a6ffd700 1 -- 172.21.15.90:0/2964305176 <== mgr.14118 v2:172.21.15.90:6800/1385551604 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe6b00611b0 con 0x7fe69c0386e0 2024-02-21T23:41:16.142 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.787+0000 7fe6b7f3c700 1 -- 172.21.15.90:0/2964305176 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7fe69c0386e0 msgr2=0x7fe69c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:16.142 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.787+0000 7fe6b7f3c700 1 --2- 172.21.15.90:0/2964305176 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7fe69c0386e0 0x7fe69c03aba0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0006fd0 tx=0x7fe6a0006e40 comp rx=0 tx=0).stop 2024-02-21T23:41:16.142 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.787+0000 7fe6b7f3c700 1 -- 172.21.15.90:0/2964305176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6b01059d0 msgr2=0x7fe6b019c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:16.142 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.787+0000 7fe6b7f3c700 1 --2- 172.21.15.90:0/2964305176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6b01059d0 0x7fe6b019c530 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac009130 tx=0x7fe6ac011ad0 comp rx=0 tx=0).stop 2024-02-21T23:41:16.142 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.787+0000 7fe6b7f3c700 1 -- 172.21.15.90:0/2964305176 shutdown_connections 2024-02-21T23:41:16.143 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.787+0000 7fe6b7f3c700 1 --2- 172.21.15.90:0/2964305176 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7fe69c0386e0 0x7fe69c03aba0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:16.143 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.787+0000 7fe6b7f3c700 1 --2- 172.21.15.90:0/2964305176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6b01059d0 0x7fe6b019c530 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:16.143 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.787+0000 7fe6b7f3c700 1 -- 172.21.15.90:0/2964305176 >> 172.21.15.90:0/2964305176 conn(0x7fe6b01000d0 msgr2=0x7fe6b0190d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:16.143 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.787+0000 7fe6b7f3c700 1 -- 172.21.15.90:0/2964305176 shutdown_connections 2024-02-21T23:41:16.143 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:15.787+0000 7fe6b7f3c700 1 -- 172.21.15.90:0/2964305176 wait complete. 2024-02-21T23:41:17.036 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:16 smithi090 ceph-mon[43385]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:41:17.036 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:16 smithi090 ceph-mon[43385]: Generating ssh key... 2024-02-21T23:41:17.037 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:16 smithi090 ceph-mon[43385]: from='mgr.14118 172.21.15.90:0/2013192540' entity='mgr.x' 2024-02-21T23:41:17.037 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:16 smithi090 ceph-mon[43385]: from='mgr.14118 172.21.15.90:0/2013192540' entity='mgr.x' 2024-02-21T23:41:17.070 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDIVpBsFA/ldrPYEoxkgLKW+56KTrEazm9Y3hRa6wwHSIavkZoUJ96kgciX9vSGf6fQithpz5fajRSU5bzciHcL2Xw5UQJedWORPBsHkY4BWmiQnYNO8Uhh+vrAB+Yo/VcsuxiP6rcZ2TCMICNK0K88vcCTQKyrErL/k38e5Py3htLNbMt61M6l+wuSrzeN2U1IWOd+EBfeeSwAVDmeByexUUkOj/4Sg/R2EtK/coJXzvdBZvwpSIhmwy3H0UaGDB73wX96+7tZprLott7k+Qtyc8tOXshEcBAFkOhcdtGNy4+b1mNwsKdccAeInngDUna1FrUbXV55Eh+KJWN99cvsTCru32U3H59q5RHCgy/luT1G6xm04Jwzc0CDJ5fobkTwKwiDJQ6MUtXQP3L4kpyMuDAwotuLmn0LgLaiOwATbao5sylb9PLonVz6OdNvWlzO0jwGbYUV8gMI9yBqUsv0lSTpaOP1oeDRD8HXZt/IRj4qEGNnP6UT5J4V360hKIs= ceph-7ed81218-d112-11ee-95bd-87774f69a715 2024-02-21T23:41:17.070 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.566+0000 7f33b86b3700 1 Processor -- start 2024-02-21T23:41:17.070 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.566+0000 7f33b86b3700 1 -- start start 2024-02-21T23:41:17.070 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.566+0000 7f33b86b3700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33b01053d0 0x7f33b01057f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:17.070 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.566+0000 7f33b86b3700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b0105dc0 con 0x7f33b01053d0 2024-02-21T23:41:17.070 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.567+0000 7f33b644f700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33b01053d0 0x7f33b01057f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:17.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.567+0000 7f33b644f700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33b01053d0 0x7f33b01057f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41026/0 (socket says 172.21.15.90:41026) 2024-02-21T23:41:17.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.567+0000 7f33b644f700 1 -- 172.21.15.90:0/2280284282 learned_addr learned my addr 172.21.15.90:0/2280284282 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:17.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.567+0000 7f33b644f700 1 -- 172.21.15.90:0/2280284282 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b01065d0 con 0x7f33b01053d0 2024-02-21T23:41:17.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.567+0000 7f33b644f700 1 --2- 172.21.15.90:0/2280284282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33b01053d0 0x7f33b01057f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f33a00098a0 tx=0x7f33a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f758ed16010a9a36 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:17.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.567+0000 7f33b544d700 1 -- 172.21.15.90:0/2280284282 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f33a0004160 con 0x7f33b01053d0 2024-02-21T23:41:17.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.567+0000 7f33b544d700 1 -- 172.21.15.90:0/2280284282 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f33a00042c0 con 0x7f33b01053d0 2024-02-21T23:41:17.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.567+0000 7f33b544d700 1 -- 172.21.15.90:0/2280284282 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f33a0011a00 con 0x7f33b01053d0 2024-02-21T23:41:17.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.568+0000 7f33b86b3700 1 -- 172.21.15.90:0/2280284282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33b01053d0 msgr2=0x7f33b01057f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:17.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.568+0000 7f33b86b3700 1 --2- 172.21.15.90:0/2280284282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33b01053d0 0x7f33b01057f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f33a00098a0 tx=0x7f33a0009bb0 comp rx=0 tx=0).stop 2024-02-21T23:41:17.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.568+0000 7f33b86b3700 1 -- 172.21.15.90:0/2280284282 shutdown_connections 2024-02-21T23:41:17.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.568+0000 7f33b86b3700 1 --2- 172.21.15.90:0/2280284282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33b01053d0 0x7f33b01057f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:17.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.568+0000 7f33b86b3700 1 -- 172.21.15.90:0/2280284282 >> 172.21.15.90:0/2280284282 conn(0x7f33b0100970 msgr2=0x7f33b0102db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:17.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.568+0000 7f33b86b3700 1 -- 172.21.15.90:0/2280284282 shutdown_connections 2024-02-21T23:41:17.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.568+0000 7f33b86b3700 1 -- 172.21.15.90:0/2280284282 wait complete. 2024-02-21T23:41:17.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.569+0000 7f33b86b3700 1 Processor -- start 2024-02-21T23:41:17.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.569+0000 7f33b86b3700 1 -- start start 2024-02-21T23:41:17.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.569+0000 7f33b86b3700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33b01053d0 0x7f33b0199040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:17.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.569+0000 7f33b86b3700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b0199580 con 0x7f33b01053d0 2024-02-21T23:41:17.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.569+0000 7f33b644f700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33b01053d0 0x7f33b0199040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:17.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.569+0000 7f33b644f700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33b01053d0 0x7f33b0199040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41034/0 (socket says 172.21.15.90:41034) 2024-02-21T23:41:17.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.569+0000 7f33b644f700 1 -- 172.21.15.90:0/2942046917 learned_addr learned my addr 172.21.15.90:0/2942046917 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:17.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.569+0000 7f33b644f700 1 -- 172.21.15.90:0/2942046917 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a0009550 con 0x7f33b01053d0 2024-02-21T23:41:17.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.569+0000 7f33b644f700 1 --2- 172.21.15.90:0/2942046917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33b01053d0 0x7f33b0199040 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f33a0009fb0 tx=0x7f33a0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:17.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.569+0000 7f33a77fe700 1 -- 172.21.15.90:0/2942046917 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f33a0003a40 con 0x7f33b01053d0 2024-02-21T23:41:17.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.569+0000 7f33a77fe700 1 -- 172.21.15.90:0/2942046917 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f33a0003ba0 con 0x7f33b01053d0 2024-02-21T23:41:17.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.569+0000 7f33a77fe700 1 -- 172.21.15.90:0/2942046917 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f33a001a750 con 0x7f33b01053d0 2024-02-21T23:41:17.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.569+0000 7f33b86b3700 1 -- 172.21.15.90:0/2942046917 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33b0199780 con 0x7f33b01053d0 2024-02-21T23:41:17.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.570+0000 7f33b86b3700 1 -- 172.21.15.90:0/2942046917 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33b0199c20 con 0x7f33b01053d0 2024-02-21T23:41:17.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.570+0000 7f33a77fe700 1 -- 172.21.15.90:0/2942046917 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) v1 ==== 45537+0+0 (secure 0 0 0) 0x7f33a001a980 con 0x7f33b01053d0 2024-02-21T23:41:17.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.570+0000 7f33b86b3700 1 -- 172.21.15.90:0/2942046917 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33b0109800 con 0x7f33b01053d0 2024-02-21T23:41:17.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.570+0000 7f33a77fe700 1 --2- 172.21.15.90:0/2942046917 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f339c038700 0x7f339c03abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:17.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.570+0000 7f33a77fe700 1 -- 172.21.15.90:0/2942046917 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f33a004c670 con 0x7f33b01053d0 2024-02-21T23:41:17.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.571+0000 7f33b5c4e700 1 --2- 172.21.15.90:0/2942046917 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f339c038700 0x7f339c03abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:17.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.571+0000 7f33b5c4e700 1 --2- 172.21.15.90:0/2942046917 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f339c038700 0x7f339c03abc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f33ac006fd0 tx=0x7f33ac006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:17.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.575+0000 7f33a77fe700 1 -- 172.21.15.90:0/2942046917 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f33a0024d40 con 0x7f33b01053d0 2024-02-21T23:41:17.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.768+0000 7f33b86b3700 1 -- 172.21.15.90:0/2942046917 --> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f33b0101c50 con 0x7f339c038700 2024-02-21T23:41:17.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.769+0000 7f33a77fe700 1 -- 172.21.15.90:0/2942046917 <== mgr.14118 v2:172.21.15.90:6800/1385551604 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f33b0101c50 con 0x7f339c038700 2024-02-21T23:41:17.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.771+0000 7f33b86b3700 1 -- 172.21.15.90:0/2942046917 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f339c038700 msgr2=0x7f339c03abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:17.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.771+0000 7f33b86b3700 1 --2- 172.21.15.90:0/2942046917 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f339c038700 0x7f339c03abc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f33ac006fd0 tx=0x7f33ac006e40 comp rx=0 tx=0).stop 2024-02-21T23:41:17.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.771+0000 7f33b86b3700 1 -- 172.21.15.90:0/2942046917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33b01053d0 msgr2=0x7f33b0199040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:17.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.771+0000 7f33b86b3700 1 --2- 172.21.15.90:0/2942046917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33b01053d0 0x7f33b0199040 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f33a0009fb0 tx=0x7f33a0011ad0 comp rx=0 tx=0).stop 2024-02-21T23:41:17.074 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.771+0000 7f33b86b3700 1 -- 172.21.15.90:0/2942046917 shutdown_connections 2024-02-21T23:41:17.074 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.771+0000 7f33b86b3700 1 --2- 172.21.15.90:0/2942046917 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f339c038700 0x7f339c03abc0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:17.074 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.771+0000 7f33b86b3700 1 --2- 172.21.15.90:0/2942046917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33b01053d0 0x7f33b0199040 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:17.074 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.771+0000 7f33b86b3700 1 -- 172.21.15.90:0/2942046917 >> 172.21.15.90:0/2942046917 conn(0x7f33b0100970 msgr2=0x7f33b0101540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:17.074 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.771+0000 7f33b86b3700 1 -- 172.21.15.90:0/2942046917 shutdown_connections 2024-02-21T23:41:17.074 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:16.771+0000 7f33b86b3700 1 -- 172.21.15.90:0/2942046917 wait complete. 2024-02-21T23:41:17.074 INFO:teuthology.orchestra.run.smithi090.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-21T23:41:17.074 INFO:teuthology.orchestra.run.smithi090.stdout:Adding key to root@localhost authorized_keys... 2024-02-21T23:41:17.074 INFO:teuthology.orchestra.run.smithi090.stdout:Adding host smithi090... 2024-02-21T23:41:17.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:17 smithi090 ceph-mon[43385]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:41:18.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:18 smithi090 ceph-mon[43385]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi090", "addr": "172.21.15.90", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:41:20.089 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout Added host 'smithi090' with addr '172.21.15.90' 2024-02-21T23:41:20.089 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.466+0000 7f716fc48700 1 Processor -- start 2024-02-21T23:41:20.089 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.466+0000 7f716fc48700 1 -- start start 2024-02-21T23:41:20.089 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.466+0000 7f716fc48700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7168106a80 0x7f7168106ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:20.089 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.466+0000 7f716fc48700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7168107470 con 0x7f7168106a80 2024-02-21T23:41:20.089 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.466+0000 7f716d9e4700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7168106a80 0x7f7168106ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:20.089 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.466+0000 7f716d9e4700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7168106a80 0x7f7168106ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41048/0 (socket says 172.21.15.90:41048) 2024-02-21T23:41:20.090 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.466+0000 7f716d9e4700 1 -- 172.21.15.90:0/2142090184 learned_addr learned my addr 172.21.15.90:0/2142090184 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:20.090 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.467+0000 7f716d9e4700 1 -- 172.21.15.90:0/2142090184 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7168105590 con 0x7f7168106a80 2024-02-21T23:41:20.090 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.467+0000 7f716d9e4700 1 --2- 172.21.15.90:0/2142090184 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7168106a80 0x7f7168106ea0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f71640098a0 tx=0x7f7164009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d59fa1766feadd4 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:20.090 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.467+0000 7f716c9e2700 1 -- 172.21.15.90:0/2142090184 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7164004160 con 0x7f7168106a80 2024-02-21T23:41:20.090 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.467+0000 7f716c9e2700 1 -- 172.21.15.90:0/2142090184 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f71640042c0 con 0x7f7168106a80 2024-02-21T23:41:20.090 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.467+0000 7f716c9e2700 1 -- 172.21.15.90:0/2142090184 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7164011a00 con 0x7f7168106a80 2024-02-21T23:41:20.090 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.467+0000 7f716fc48700 1 -- 172.21.15.90:0/2142090184 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7168106a80 msgr2=0x7f7168106ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:20.090 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.467+0000 7f716fc48700 1 --2- 172.21.15.90:0/2142090184 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7168106a80 0x7f7168106ea0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f71640098a0 tx=0x7f7164009bb0 comp rx=0 tx=0).stop 2024-02-21T23:41:20.090 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.467+0000 7f716fc48700 1 -- 172.21.15.90:0/2142090184 shutdown_connections 2024-02-21T23:41:20.090 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.467+0000 7f716fc48700 1 --2- 172.21.15.90:0/2142090184 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7168106a80 0x7f7168106ea0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:20.090 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.467+0000 7f716fc48700 1 -- 172.21.15.90:0/2142090184 >> 172.21.15.90:0/2142090184 conn(0x7f71681000a0 msgr2=0x7f7168102500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:20.090 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.468+0000 7f716fc48700 1 -- 172.21.15.90:0/2142090184 shutdown_connections 2024-02-21T23:41:20.090 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.468+0000 7f716fc48700 1 -- 172.21.15.90:0/2142090184 wait complete. 2024-02-21T23:41:20.090 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.468+0000 7f716fc48700 1 Processor -- start 2024-02-21T23:41:20.091 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.468+0000 7f716fc48700 1 -- start start 2024-02-21T23:41:20.091 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.468+0000 7f716fc48700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7168106a80 0x7f7168198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:20.091 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.468+0000 7f716fc48700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71681986b0 con 0x7f7168106a80 2024-02-21T23:41:20.091 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.468+0000 7f716d9e4700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7168106a80 0x7f7168198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:20.091 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.468+0000 7f716d9e4700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7168106a80 0x7f7168198170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41060/0 (socket says 172.21.15.90:41060) 2024-02-21T23:41:20.091 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.468+0000 7f716d9e4700 1 -- 172.21.15.90:0/302639214 learned_addr learned my addr 172.21.15.90:0/302639214 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:20.091 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.469+0000 7f716d9e4700 1 -- 172.21.15.90:0/302639214 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7164009550 con 0x7f7168106a80 2024-02-21T23:41:20.091 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.469+0000 7f716d9e4700 1 --2- 172.21.15.90:0/302639214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7168106a80 0x7f7168198170 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7164009130 tx=0x7f7164011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:20.091 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.469+0000 7f715effd700 1 -- 172.21.15.90:0/302639214 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f71640038b0 con 0x7f7168106a80 2024-02-21T23:41:20.091 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.469+0000 7f715effd700 1 -- 172.21.15.90:0/302639214 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7164003a10 con 0x7f7168106a80 2024-02-21T23:41:20.091 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.469+0000 7f716fc48700 1 -- 172.21.15.90:0/302639214 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71681988b0 con 0x7f7168106a80 2024-02-21T23:41:20.091 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.469+0000 7f715effd700 1 -- 172.21.15.90:0/302639214 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7164019550 con 0x7f7168106a80 2024-02-21T23:41:20.091 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.469+0000 7f716fc48700 1 -- 172.21.15.90:0/302639214 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7168198d50 con 0x7f7168106a80 2024-02-21T23:41:20.091 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.470+0000 7f715effd700 1 -- 172.21.15.90:0/302639214 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) v1 ==== 45537+0+0 (secure 0 0 0) 0x7f716401a430 con 0x7f7168106a80 2024-02-21T23:41:20.091 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.470+0000 7f716fc48700 1 -- 172.21.15.90:0/302639214 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7168191c30 con 0x7f7168106a80 2024-02-21T23:41:20.092 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.470+0000 7f715effd700 1 --2- 172.21.15.90:0/302639214 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f715403caa0 0x7f715403ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:20.092 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.470+0000 7f715effd700 1 -- 172.21.15.90:0/302639214 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f716404c7c0 con 0x7f7168106a80 2024-02-21T23:41:20.095 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.470+0000 7f716d1e3700 1 --2- 172.21.15.90:0/302639214 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f715403caa0 0x7f715403ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:20.096 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.471+0000 7f716d1e3700 1 --2- 172.21.15.90:0/302639214 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f715403caa0 0x7f715403ef60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7158006fd0 tx=0x7f7158006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:20.096 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.475+0000 7f715effd700 1 -- 172.21.15.90:0/302639214 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f7164023b80 con 0x7f7168106a80 2024-02-21T23:41:20.096 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:17.671+0000 7f716fc48700 1 -- 172.21.15.90:0/302639214 --> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi090", "addr": "172.21.15.90", "target": ["mon-mgr", ""]}) v1 -- 0x7f71680611b0 con 0x7f715403caa0 2024-02-21T23:41:20.096 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:19.787+0000 7f715effd700 1 -- 172.21.15.90:0/302639214 <== mgr.14118 v2:172.21.15.90:6800/1385551604 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f71680611b0 con 0x7f715403caa0 2024-02-21T23:41:20.096 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:19.789+0000 7f716fc48700 1 -- 172.21.15.90:0/302639214 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f715403caa0 msgr2=0x7f715403ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:20.096 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:19.789+0000 7f716fc48700 1 --2- 172.21.15.90:0/302639214 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f715403caa0 0x7f715403ef60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7158006fd0 tx=0x7f7158006e40 comp rx=0 tx=0).stop 2024-02-21T23:41:20.096 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:19.789+0000 7f716fc48700 1 -- 172.21.15.90:0/302639214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7168106a80 msgr2=0x7f7168198170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:20.096 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:19.789+0000 7f716fc48700 1 --2- 172.21.15.90:0/302639214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7168106a80 0x7f7168198170 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7164009130 tx=0x7f7164011ad0 comp rx=0 tx=0).stop 2024-02-21T23:41:20.096 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:19.789+0000 7f716fc48700 1 -- 172.21.15.90:0/302639214 shutdown_connections 2024-02-21T23:41:20.096 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:19.789+0000 7f716fc48700 1 --2- 172.21.15.90:0/302639214 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f715403caa0 0x7f715403ef60 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:20.096 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:19.789+0000 7f716fc48700 1 --2- 172.21.15.90:0/302639214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7168106a80 0x7f7168198170 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:20.096 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:19.789+0000 7f716fc48700 1 -- 172.21.15.90:0/302639214 >> 172.21.15.90:0/302639214 conn(0x7f71681000a0 msgr2=0x7f716818ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:20.096 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:19.789+0000 7f716fc48700 1 -- 172.21.15.90:0/302639214 shutdown_connections 2024-02-21T23:41:20.096 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:19.789+0000 7f716fc48700 1 -- 172.21.15.90:0/302639214 wait complete. 2024-02-21T23:41:20.097 INFO:teuthology.orchestra.run.smithi090.stdout:Deploying unmanaged mon service... 2024-02-21T23:41:20.097 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:19 smithi090 ceph-mon[43385]: Deploying cephadm binary to smithi090 2024-02-21T23:41:21.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:20 smithi090 ceph-mon[43385]: from='mgr.14118 172.21.15.90:0/2013192540' entity='mgr.x' 2024-02-21T23:41:21.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:20 smithi090 ceph-mon[43385]: Added host smithi090 2024-02-21T23:41:21.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:20 smithi090 ceph-mon[43385]: from='mgr.14118 172.21.15.90:0/2013192540' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:41:21.424 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-02-21T23:41:21.424 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.589+0000 7f3cc3e3e700 1 Processor -- start 2024-02-21T23:41:21.424 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.589+0000 7f3cc3e3e700 1 -- start start 2024-02-21T23:41:21.425 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.589+0000 7f3cc3e3e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3cbc106f40 0x7f3cbc109330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:21.425 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.589+0000 7f3cc3e3e700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cbc079f60 con 0x7f3cbc106f40 2024-02-21T23:41:21.425 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.590+0000 7f3cc1bda700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3cbc106f40 0x7f3cbc109330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:21.425 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.590+0000 7f3cc1bda700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3cbc106f40 0x7f3cbc109330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57282/0 (socket says 172.21.15.90:57282) 2024-02-21T23:41:21.425 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.590+0000 7f3cc1bda700 1 -- 172.21.15.90:0/1604758165 learned_addr learned my addr 172.21.15.90:0/1604758165 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:21.425 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.590+0000 7f3cc1bda700 1 -- 172.21.15.90:0/1604758165 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cbc109870 con 0x7f3cbc106f40 2024-02-21T23:41:21.425 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.590+0000 7f3cc1bda700 1 --2- 172.21.15.90:0/1604758165 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3cbc106f40 0x7f3cbc109330 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3cac0098a0 tx=0x7f3cac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3057cddfe7bbd7a3 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:21.425 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.590+0000 7f3cc0bd8700 1 -- 172.21.15.90:0/1604758165 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3cac004160 con 0x7f3cbc106f40 2024-02-21T23:41:21.425 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.590+0000 7f3cc0bd8700 1 -- 172.21.15.90:0/1604758165 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3cac0042c0 con 0x7f3cbc106f40 2024-02-21T23:41:21.425 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.590+0000 7f3cc0bd8700 1 -- 172.21.15.90:0/1604758165 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3cac011a00 con 0x7f3cbc106f40 2024-02-21T23:41:21.425 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.591+0000 7f3cc3e3e700 1 -- 172.21.15.90:0/1604758165 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3cbc106f40 msgr2=0x7f3cbc109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:21.426 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.591+0000 7f3cc3e3e700 1 --2- 172.21.15.90:0/1604758165 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3cbc106f40 0x7f3cbc109330 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3cac0098a0 tx=0x7f3cac009bb0 comp rx=0 tx=0).stop 2024-02-21T23:41:21.426 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.591+0000 7f3cc3e3e700 1 -- 172.21.15.90:0/1604758165 shutdown_connections 2024-02-21T23:41:21.426 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.591+0000 7f3cc3e3e700 1 --2- 172.21.15.90:0/1604758165 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3cbc106f40 0x7f3cbc109330 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:21.426 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.591+0000 7f3cc3e3e700 1 -- 172.21.15.90:0/1604758165 >> 172.21.15.90:0/1604758165 conn(0x7f3cbc100970 msgr2=0x7f3cbc102d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:21.426 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.591+0000 7f3cc3e3e700 1 -- 172.21.15.90:0/1604758165 shutdown_connections 2024-02-21T23:41:21.426 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.591+0000 7f3cc3e3e700 1 -- 172.21.15.90:0/1604758165 wait complete. 2024-02-21T23:41:21.426 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.592+0000 7f3cc3e3e700 1 Processor -- start 2024-02-21T23:41:21.426 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.592+0000 7f3cc3e3e700 1 -- start start 2024-02-21T23:41:21.426 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.592+0000 7f3cc3e3e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3cbc106f40 0x7f3cbc19a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:21.426 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.592+0000 7f3cc3e3e700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cbc19a850 con 0x7f3cbc106f40 2024-02-21T23:41:21.427 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.592+0000 7f3cc1bda700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3cbc106f40 0x7f3cbc19a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:21.427 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.592+0000 7f3cc1bda700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3cbc106f40 0x7f3cbc19a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57290/0 (socket says 172.21.15.90:57290) 2024-02-21T23:41:21.427 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.592+0000 7f3cc1bda700 1 -- 172.21.15.90:0/2926728654 learned_addr learned my addr 172.21.15.90:0/2926728654 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:21.427 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.592+0000 7f3cc1bda700 1 -- 172.21.15.90:0/2926728654 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cac009550 con 0x7f3cbc106f40 2024-02-21T23:41:21.427 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.592+0000 7f3cc1bda700 1 --2- 172.21.15.90:0/2926728654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3cbc106f40 0x7f3cbc19a310 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3cac009130 tx=0x7f3cac011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:21.427 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.592+0000 7f3cb2ffd700 1 -- 172.21.15.90:0/2926728654 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3cac0038b0 con 0x7f3cbc106f40 2024-02-21T23:41:21.427 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.592+0000 7f3cb2ffd700 1 -- 172.21.15.90:0/2926728654 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3cac003a10 con 0x7f3cbc106f40 2024-02-21T23:41:21.427 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.592+0000 7f3cc3e3e700 1 -- 172.21.15.90:0/2926728654 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cbc19aa50 con 0x7f3cbc106f40 2024-02-21T23:41:21.427 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.593+0000 7f3cb2ffd700 1 -- 172.21.15.90:0/2926728654 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3cac019550 con 0x7f3cbc106f40 2024-02-21T23:41:21.427 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.593+0000 7f3cc3e3e700 1 -- 172.21.15.90:0/2926728654 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cbc19aef0 con 0x7f3cbc106f40 2024-02-21T23:41:21.427 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.593+0000 7f3cb2ffd700 1 -- 172.21.15.90:0/2926728654 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) v1 ==== 45537+0+0 (secure 0 0 0) 0x7f3cac019850 con 0x7f3cbc106f40 2024-02-21T23:41:21.427 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.593+0000 7f3cb2ffd700 1 --2- 172.21.15.90:0/2926728654 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f3ca8038690 0x7f3ca803ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:21.428 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.593+0000 7f3cb2ffd700 1 -- 172.21.15.90:0/2926728654 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f3cac04c7d0 con 0x7f3cbc106f40 2024-02-21T23:41:21.428 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.593+0000 7f3cc3e3e700 1 -- 172.21.15.90:0/2926728654 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cbc0623a0 con 0x7f3cbc106f40 2024-02-21T23:41:21.428 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.593+0000 7f3cc13d9700 1 --2- 172.21.15.90:0/2926728654 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f3ca8038690 0x7f3ca803ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:21.428 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.594+0000 7f3cc13d9700 1 --2- 172.21.15.90:0/2926728654 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f3ca8038690 0x7f3ca803ab50 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8006fd0 tx=0x7f3cb8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:21.428 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.598+0000 7f3cb2ffd700 1 -- 172.21.15.90:0/2926728654 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f3cac019b50 con 0x7f3cbc106f40 2024-02-21T23:41:21.428 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.813+0000 7f3cc3e3e700 1 -- 172.21.15.90:0/2926728654 --> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f3cbc101c50 con 0x7f3ca8038690 2024-02-21T23:41:21.428 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.818+0000 7f3cb2ffd700 1 -- 172.21.15.90:0/2926728654 <== mgr.14118 v2:172.21.15.90:6800/1385551604 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f3cbc101c50 con 0x7f3ca8038690 2024-02-21T23:41:21.428 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.820+0000 7f3cc3e3e700 1 -- 172.21.15.90:0/2926728654 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f3ca8038690 msgr2=0x7f3ca803ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:21.428 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.820+0000 7f3cc3e3e700 1 --2- 172.21.15.90:0/2926728654 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f3ca8038690 0x7f3ca803ab50 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8006fd0 tx=0x7f3cb8006e40 comp rx=0 tx=0).stop 2024-02-21T23:41:21.428 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.820+0000 7f3cc3e3e700 1 -- 172.21.15.90:0/2926728654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3cbc106f40 msgr2=0x7f3cbc19a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:21.428 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.820+0000 7f3cc3e3e700 1 --2- 172.21.15.90:0/2926728654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3cbc106f40 0x7f3cbc19a310 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3cac009130 tx=0x7f3cac011ad0 comp rx=0 tx=0).stop 2024-02-21T23:41:21.428 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.820+0000 7f3cc3e3e700 1 -- 172.21.15.90:0/2926728654 shutdown_connections 2024-02-21T23:41:21.429 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.820+0000 7f3cc3e3e700 1 --2- 172.21.15.90:0/2926728654 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f3ca8038690 0x7f3ca803ab50 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:21.429 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.820+0000 7f3cc3e3e700 1 --2- 172.21.15.90:0/2926728654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3cbc106f40 0x7f3cbc19a310 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:21.429 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.820+0000 7f3cc3e3e700 1 -- 172.21.15.90:0/2926728654 >> 172.21.15.90:0/2926728654 conn(0x7f3cbc100970 msgr2=0x7f3cbc101540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:21.429 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.820+0000 7f3cc3e3e700 1 -- 172.21.15.90:0/2926728654 shutdown_connections 2024-02-21T23:41:21.429 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:20.820+0000 7f3cc3e3e700 1 -- 172.21.15.90:0/2926728654 wait complete. 2024-02-21T23:41:21.429 INFO:teuthology.orchestra.run.smithi090.stdout:Deploying unmanaged mgr service... 2024-02-21T23:41:21.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:21 smithi090 ceph-mon[43385]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:41:21.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:21 smithi090 ceph-mon[43385]: Saving service mon spec with placement count:5 2024-02-21T23:41:21.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:21 smithi090 ceph-mon[43385]: from='mgr.14118 172.21.15.90:0/2013192540' entity='mgr.x' 2024-02-21T23:41:21.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:21 smithi090 ceph-mon[43385]: from='mgr.14118 172.21.15.90:0/2013192540' entity='mgr.x' 2024-02-21T23:41:21.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:21 smithi090 ceph-mon[43385]: from='mgr.14118 172.21.15.90:0/2013192540' entity='mgr.x' 2024-02-21T23:41:22.497 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-02-21T23:41:22.497 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.802+0000 7f901d9dd700 1 Processor -- start 2024-02-21T23:41:22.497 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.802+0000 7f901d9dd700 1 -- start start 2024-02-21T23:41:22.497 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.802+0000 7f901d9dd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9018104d70 0x7f9018105190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:22.497 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.802+0000 7f901d9dd700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9018079f20 con 0x7f9018104d70 2024-02-21T23:41:22.497 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.803+0000 7f9016ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9018104d70 0x7f9018105190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:22.497 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.803+0000 7f9016ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9018104d70 0x7f9018105190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57304/0 (socket says 172.21.15.90:57304) 2024-02-21T23:41:22.497 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.803+0000 7f9016ffd700 1 -- 172.21.15.90:0/2774628612 learned_addr learned my addr 172.21.15.90:0/2774628612 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:22.497 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.803+0000 7f9016ffd700 1 -- 172.21.15.90:0/2774628612 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90181056d0 con 0x7f9018104d70 2024-02-21T23:41:22.498 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.803+0000 7f9016ffd700 1 --2- 172.21.15.90:0/2774628612 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9018104d70 0x7f9018105190 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f90080098a0 tx=0x7f9008009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5ecc98fa19a6ff6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:22.498 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.803+0000 7f9015ffb700 1 -- 172.21.15.90:0/2774628612 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9008004160 con 0x7f9018104d70 2024-02-21T23:41:22.498 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.803+0000 7f9015ffb700 1 -- 172.21.15.90:0/2774628612 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f90080042c0 con 0x7f9018104d70 2024-02-21T23:41:22.498 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.803+0000 7f9015ffb700 1 -- 172.21.15.90:0/2774628612 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f90080119b0 con 0x7f9018104d70 2024-02-21T23:41:22.498 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.804+0000 7f901d9dd700 1 -- 172.21.15.90:0/2774628612 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9018104d70 msgr2=0x7f9018105190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:22.498 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.804+0000 7f901d9dd700 1 --2- 172.21.15.90:0/2774628612 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9018104d70 0x7f9018105190 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f90080098a0 tx=0x7f9008009bb0 comp rx=0 tx=0).stop 2024-02-21T23:41:22.498 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.804+0000 7f901d9dd700 1 -- 172.21.15.90:0/2774628612 shutdown_connections 2024-02-21T23:41:22.498 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.804+0000 7f901d9dd700 1 --2- 172.21.15.90:0/2774628612 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9018104d70 0x7f9018105190 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:22.498 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.804+0000 7f901d9dd700 1 -- 172.21.15.90:0/2774628612 >> 172.21.15.90:0/2774628612 conn(0x7f9018100970 msgr2=0x7f9018102d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:22.498 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.804+0000 7f901d9dd700 1 -- 172.21.15.90:0/2774628612 shutdown_connections 2024-02-21T23:41:22.498 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.804+0000 7f901d9dd700 1 -- 172.21.15.90:0/2774628612 wait complete. 2024-02-21T23:41:22.498 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.804+0000 7f901d9dd700 1 Processor -- start 2024-02-21T23:41:22.498 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.805+0000 7f901d9dd700 1 -- start start 2024-02-21T23:41:22.498 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.805+0000 7f901d9dd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9018104d70 0x7f901819c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:22.499 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.805+0000 7f901d9dd700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f901819ca40 con 0x7f9018104d70 2024-02-21T23:41:22.499 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.805+0000 7f9016ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9018104d70 0x7f901819c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:22.499 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.805+0000 7f9016ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9018104d70 0x7f901819c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57318/0 (socket says 172.21.15.90:57318) 2024-02-21T23:41:22.499 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.805+0000 7f9016ffd700 1 -- 172.21.15.90:0/161065303 learned_addr learned my addr 172.21.15.90:0/161065303 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:22.499 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.805+0000 7f9016ffd700 1 -- 172.21.15.90:0/161065303 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9008009550 con 0x7f9018104d70 2024-02-21T23:41:22.499 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.805+0000 7f9016ffd700 1 --2- 172.21.15.90:0/161065303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9018104d70 0x7f901819c500 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9008011980 tx=0x7f9008011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:22.499 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.805+0000 7f901c9db700 1 -- 172.21.15.90:0/161065303 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9008003af0 con 0x7f9018104d70 2024-02-21T23:41:22.499 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.806+0000 7f901d9dd700 1 -- 172.21.15.90:0/161065303 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f901819cc40 con 0x7f9018104d70 2024-02-21T23:41:22.499 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.806+0000 7f901c9db700 1 -- 172.21.15.90:0/161065303 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9008003c50 con 0x7f9018104d70 2024-02-21T23:41:22.499 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.806+0000 7f901c9db700 1 -- 172.21.15.90:0/161065303 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9008019760 con 0x7f9018104d70 2024-02-21T23:41:22.499 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.806+0000 7f901d9dd700 1 -- 172.21.15.90:0/161065303 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f901819d0e0 con 0x7f9018104d70 2024-02-21T23:41:22.499 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.806+0000 7f901c9db700 1 -- 172.21.15.90:0/161065303 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) v1 ==== 45537+0+0 (secure 0 0 0) 0x7f90080199a0 con 0x7f9018104d70 2024-02-21T23:41:22.499 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.806+0000 7f901d9dd700 1 -- 172.21.15.90:0/161065303 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90181961d0 con 0x7f9018104d70 2024-02-21T23:41:22.499 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.807+0000 7f901c9db700 1 --2- 172.21.15.90:0/161065303 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f9000038700 0x7f900003abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:22.500 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.807+0000 7f901c9db700 1 -- 172.21.15.90:0/161065303 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f900804ca10 con 0x7f9018104d70 2024-02-21T23:41:22.500 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.807+0000 7f90167fc700 1 --2- 172.21.15.90:0/161065303 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f9000038700 0x7f900003abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:22.500 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.807+0000 7f90167fc700 1 --2- 172.21.15.90:0/161065303 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f9000038700 0x7f900003abc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f900c006fd0 tx=0x7f900c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:22.500 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:21.811+0000 7f901c9db700 1 -- 172.21.15.90:0/161065303 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f9008018bd0 con 0x7f9018104d70 2024-02-21T23:41:22.500 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:22.036+0000 7f901d9dd700 1 -- 172.21.15.90:0/161065303 --> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f90180611b0 con 0x7f9000038700 2024-02-21T23:41:22.500 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:22.040+0000 7f901c9db700 1 -- 172.21.15.90:0/161065303 <== mgr.14118 v2:172.21.15.90:6800/1385551604 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f90180611b0 con 0x7f9000038700 2024-02-21T23:41:22.500 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:22.043+0000 7f901d9dd700 1 -- 172.21.15.90:0/161065303 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f9000038700 msgr2=0x7f900003abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:22.500 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:22.043+0000 7f901d9dd700 1 --2- 172.21.15.90:0/161065303 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f9000038700 0x7f900003abc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f900c006fd0 tx=0x7f900c006e40 comp rx=0 tx=0).stop 2024-02-21T23:41:22.500 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:22.043+0000 7f901d9dd700 1 -- 172.21.15.90:0/161065303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9018104d70 msgr2=0x7f901819c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:22.500 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:22.043+0000 7f901d9dd700 1 --2- 172.21.15.90:0/161065303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9018104d70 0x7f901819c500 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9008011980 tx=0x7f9008011a60 comp rx=0 tx=0).stop 2024-02-21T23:41:22.500 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:22.043+0000 7f901d9dd700 1 -- 172.21.15.90:0/161065303 shutdown_connections 2024-02-21T23:41:22.500 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:22.043+0000 7f901d9dd700 1 --2- 172.21.15.90:0/161065303 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f9000038700 0x7f900003abc0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:22.500 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:22.043+0000 7f901d9dd700 1 --2- 172.21.15.90:0/161065303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9018104d70 0x7f901819c500 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:22.501 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:22.043+0000 7f901d9dd700 1 -- 172.21.15.90:0/161065303 >> 172.21.15.90:0/161065303 conn(0x7f9018100970 msgr2=0x7f9018193310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:22.501 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:22.043+0000 7f901d9dd700 1 -- 172.21.15.90:0/161065303 shutdown_connections 2024-02-21T23:41:22.501 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:22.043+0000 7f901d9dd700 1 -- 172.21.15.90:0/161065303 wait complete. 2024-02-21T23:41:23.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:23 smithi090 ceph-mon[43385]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:41:23.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:23 smithi090 ceph-mon[43385]: Saving service mgr spec with placement count:2 2024-02-21T23:41:23.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:23 smithi090 ceph-mon[43385]: from='mgr.14118 172.21.15.90:0/2013192540' entity='mgr.x' 2024-02-21T23:41:23.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:23 smithi090 ceph-mon[43385]: from='mgr.14118 172.21.15.90:0/2013192540' entity='mgr.x' 2024-02-21T23:41:23.731 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.199+0000 7f01ae4bc700 1 Processor -- start 2024-02-21T23:41:23.731 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.199+0000 7f01ae4bc700 1 -- start start 2024-02-21T23:41:23.731 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.199+0000 7f01ae4bc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01a8106a50 0x7f01a8106e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:23.731 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.199+0000 7f01ae4bc700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a8107440 con 0x7f01a8106a50 2024-02-21T23:41:23.731 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.199+0000 7f01a7fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01a8106a50 0x7f01a8106e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:23.731 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.199+0000 7f01a7fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01a8106a50 0x7f01a8106e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57334/0 (socket says 172.21.15.90:57334) 2024-02-21T23:41:23.731 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.199+0000 7f01a7fff700 1 -- 172.21.15.90:0/2532293752 learned_addr learned my addr 172.21.15.90:0/2532293752 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:23.731 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.200+0000 7f01a7fff700 1 -- 172.21.15.90:0/2532293752 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a8105510 con 0x7f01a8106a50 2024-02-21T23:41:23.732 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.200+0000 7f01a7fff700 1 --2- 172.21.15.90:0/2532293752 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01a8106a50 0x7f01a8106e70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f019800bb40 tx=0x7f019800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=459996ceb692912e server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:23.732 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.200+0000 7f01a6ffd700 1 -- 172.21.15.90:0/2532293752 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f019800dbe0 con 0x7f01a8106a50 2024-02-21T23:41:23.732 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.200+0000 7f01a6ffd700 1 -- 172.21.15.90:0/2532293752 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f01980044f0 con 0x7f01a8106a50 2024-02-21T23:41:23.732 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.200+0000 7f01a6ffd700 1 -- 172.21.15.90:0/2532293752 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0198003e60 con 0x7f01a8106a50 2024-02-21T23:41:23.732 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.201+0000 7f01ae4bc700 1 -- 172.21.15.90:0/2532293752 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01a8106a50 msgr2=0x7f01a8106e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:23.732 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.201+0000 7f01ae4bc700 1 --2- 172.21.15.90:0/2532293752 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01a8106a50 0x7f01a8106e70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f019800bb40 tx=0x7f019800be50 comp rx=0 tx=0).stop 2024-02-21T23:41:23.732 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.201+0000 7f01ae4bc700 1 -- 172.21.15.90:0/2532293752 shutdown_connections 2024-02-21T23:41:23.732 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.201+0000 7f01ae4bc700 1 --2- 172.21.15.90:0/2532293752 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01a8106a50 0x7f01a8106e70 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:23.732 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.201+0000 7f01ae4bc700 1 -- 172.21.15.90:0/2532293752 >> 172.21.15.90:0/2532293752 conn(0x7f01a81000a0 msgr2=0x7f01a8102500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:23.732 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.201+0000 7f01ae4bc700 1 -- 172.21.15.90:0/2532293752 shutdown_connections 2024-02-21T23:41:23.733 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.201+0000 7f01ae4bc700 1 -- 172.21.15.90:0/2532293752 wait complete. 2024-02-21T23:41:23.733 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.202+0000 7f01ae4bc700 1 Processor -- start 2024-02-21T23:41:23.733 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.202+0000 7f01ae4bc700 1 -- start start 2024-02-21T23:41:23.733 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.202+0000 7f01ae4bc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01a8106a50 0x7f01a819c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:23.733 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.202+0000 7f01ae4bc700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a819cad0 con 0x7f01a8106a50 2024-02-21T23:41:23.733 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.202+0000 7f01a7fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01a8106a50 0x7f01a819c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:23.733 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.202+0000 7f01a7fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01a8106a50 0x7f01a819c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57342/0 (socket says 172.21.15.90:57342) 2024-02-21T23:41:23.734 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.202+0000 7f01a7fff700 1 -- 172.21.15.90:0/1849453566 learned_addr learned my addr 172.21.15.90:0/1849453566 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:23.734 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.203+0000 7f01a7fff700 1 -- 172.21.15.90:0/1849453566 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f019800b7f0 con 0x7f01a8106a50 2024-02-21T23:41:23.734 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.203+0000 7f01a7fff700 1 --2- 172.21.15.90:0/1849453566 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01a8106a50 0x7f01a819c590 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0198003bf0 tx=0x7f0198003fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:23.734 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.203+0000 7f01a57fa700 1 -- 172.21.15.90:0/1849453566 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0198010500 con 0x7f01a8106a50 2024-02-21T23:41:23.734 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.203+0000 7f01a57fa700 1 -- 172.21.15.90:0/1849453566 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0198010b20 con 0x7f01a8106a50 2024-02-21T23:41:23.734 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.203+0000 7f01ae4bc700 1 -- 172.21.15.90:0/1849453566 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01a819ccd0 con 0x7f01a8106a50 2024-02-21T23:41:23.735 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.203+0000 7f01a57fa700 1 -- 172.21.15.90:0/1849453566 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0198021ad0 con 0x7f01a8106a50 2024-02-21T23:41:23.735 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.203+0000 7f01ae4bc700 1 -- 172.21.15.90:0/1849453566 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01a819d110 con 0x7f01a8106a50 2024-02-21T23:41:23.735 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.204+0000 7f01a57fa700 1 -- 172.21.15.90:0/1849453566 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) v1 ==== 45537+0+0 (secure 0 0 0) 0x7f0198021d50 con 0x7f01a8106a50 2024-02-21T23:41:23.735 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.204+0000 7f01a57fa700 1 --2- 172.21.15.90:0/1849453566 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f0190038700 0x7f019003abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:23.735 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.204+0000 7f01ae4bc700 1 -- 172.21.15.90:0/1849453566 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01a8104400 con 0x7f01a8106a50 2024-02-21T23:41:23.735 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.204+0000 7f01a57fa700 1 -- 172.21.15.90:0/1849453566 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f019804c880 con 0x7f01a8106a50 2024-02-21T23:41:23.735 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.204+0000 7f01a77fe700 1 --2- 172.21.15.90:0/1849453566 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f0190038700 0x7f019003abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:23.736 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.205+0000 7f01a77fe700 1 --2- 172.21.15.90:0/1849453566 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f0190038700 0x7f019003abc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f019c006fd0 tx=0x7f019c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:23.736 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.209+0000 7f01a57fa700 1 -- 172.21.15.90:0/1849453566 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f0198044e60 con 0x7f01a8106a50 2024-02-21T23:41:23.736 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.413+0000 7f01ae4bc700 1 -- 172.21.15.90:0/1849453566 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f01a81918c0 con 0x7f01a8106a50 2024-02-21T23:41:23.736 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.420+0000 7f01a57fa700 1 -- 172.21.15.90:0/1849453566 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f0198006d80 con 0x7f01a8106a50 2024-02-21T23:41:23.736 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.422+0000 7f01ae4bc700 1 -- 172.21.15.90:0/1849453566 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f0190038700 msgr2=0x7f019003abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:23.736 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.422+0000 7f01ae4bc700 1 --2- 172.21.15.90:0/1849453566 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f0190038700 0x7f019003abc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f019c006fd0 tx=0x7f019c006e40 comp rx=0 tx=0).stop 2024-02-21T23:41:23.736 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.422+0000 7f01ae4bc700 1 -- 172.21.15.90:0/1849453566 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01a8106a50 msgr2=0x7f01a819c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:23.736 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.422+0000 7f01ae4bc700 1 --2- 172.21.15.90:0/1849453566 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01a8106a50 0x7f01a819c590 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0198003bf0 tx=0x7f0198003fb0 comp rx=0 tx=0).stop 2024-02-21T23:41:23.737 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.422+0000 7f01ae4bc700 1 -- 172.21.15.90:0/1849453566 shutdown_connections 2024-02-21T23:41:23.737 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.422+0000 7f01ae4bc700 1 --2- 172.21.15.90:0/1849453566 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f0190038700 0x7f019003abc0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:23.737 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.422+0000 7f01ae4bc700 1 --2- 172.21.15.90:0/1849453566 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01a8106a50 0x7f01a819c590 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:23.737 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.422+0000 7f01ae4bc700 1 -- 172.21.15.90:0/1849453566 >> 172.21.15.90:0/1849453566 conn(0x7f01a81000a0 msgr2=0x7f01a818e980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:23.737 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.422+0000 7f01ae4bc700 1 -- 172.21.15.90:0/1849453566 shutdown_connections 2024-02-21T23:41:23.737 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:23.422+0000 7f01ae4bc700 1 -- 172.21.15.90:0/1849453566 wait complete. 2024-02-21T23:41:24.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:24 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/1849453566' entity='client.admin' 2024-02-21T23:41:24.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:24 smithi090 ceph-mon[43385]: from='mgr.14118 172.21.15.90:0/2013192540' entity='mgr.x' 2024-02-21T23:41:25.273 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.386+0000 7f9c18e40700 1 Processor -- start 2024-02-21T23:41:25.273 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.386+0000 7f9c18e40700 1 -- start start 2024-02-21T23:41:25.273 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.386+0000 7f9c18e40700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c14106f60 0x7f9c14109350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:25.273 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.386+0000 7f9c18e40700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c14079f60 con 0x7f9c14106f60 2024-02-21T23:41:25.273 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.387+0000 7f9c1259c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c14106f60 0x7f9c14109350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:25.273 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.387+0000 7f9c1259c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c14106f60 0x7f9c14109350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57358/0 (socket says 172.21.15.90:57358) 2024-02-21T23:41:25.274 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.387+0000 7f9c1259c700 1 -- 172.21.15.90:0/2964967882 learned_addr learned my addr 172.21.15.90:0/2964967882 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:25.274 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.387+0000 7f9c1259c700 1 -- 172.21.15.90:0/2964967882 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c14109890 con 0x7f9c14106f60 2024-02-21T23:41:25.274 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.387+0000 7f9c1259c700 1 --2- 172.21.15.90:0/2964967882 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c14106f60 0x7f9c14109350 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9c080098a0 tx=0x7f9c08009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e54a31273ebf863 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:25.274 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.387+0000 7f9c1159a700 1 -- 172.21.15.90:0/2964967882 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9c08004160 con 0x7f9c14106f60 2024-02-21T23:41:25.274 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.387+0000 7f9c1159a700 1 -- 172.21.15.90:0/2964967882 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9c080042c0 con 0x7f9c14106f60 2024-02-21T23:41:25.274 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.387+0000 7f9c1159a700 1 -- 172.21.15.90:0/2964967882 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9c080119b0 con 0x7f9c14106f60 2024-02-21T23:41:25.274 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.388+0000 7f9c18e40700 1 -- 172.21.15.90:0/2964967882 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c14106f60 msgr2=0x7f9c14109350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:25.274 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.388+0000 7f9c18e40700 1 --2- 172.21.15.90:0/2964967882 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c14106f60 0x7f9c14109350 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9c080098a0 tx=0x7f9c08009bb0 comp rx=0 tx=0).stop 2024-02-21T23:41:25.274 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.388+0000 7f9c18e40700 1 -- 172.21.15.90:0/2964967882 shutdown_connections 2024-02-21T23:41:25.274 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.388+0000 7f9c18e40700 1 --2- 172.21.15.90:0/2964967882 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c14106f60 0x7f9c14109350 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:25.274 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.388+0000 7f9c18e40700 1 -- 172.21.15.90:0/2964967882 >> 172.21.15.90:0/2964967882 conn(0x7f9c14100970 msgr2=0x7f9c14102d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:25.275 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.388+0000 7f9c18e40700 1 -- 172.21.15.90:0/2964967882 shutdown_connections 2024-02-21T23:41:25.275 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.388+0000 7f9c18e40700 1 -- 172.21.15.90:0/2964967882 wait complete. 2024-02-21T23:41:25.275 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.389+0000 7f9c18e40700 1 Processor -- start 2024-02-21T23:41:25.275 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.389+0000 7f9c18e40700 1 -- start start 2024-02-21T23:41:25.275 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.389+0000 7f9c18e40700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c14106f60 0x7f9c1419a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:25.275 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.389+0000 7f9c18e40700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1419a800 con 0x7f9c14106f60 2024-02-21T23:41:25.275 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.389+0000 7f9c1259c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c14106f60 0x7f9c1419a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:25.275 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.389+0000 7f9c1259c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c14106f60 0x7f9c1419a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57372/0 (socket says 172.21.15.90:57372) 2024-02-21T23:41:25.275 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.389+0000 7f9c1259c700 1 -- 172.21.15.90:0/1094613857 learned_addr learned my addr 172.21.15.90:0/1094613857 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:25.275 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.389+0000 7f9c1259c700 1 -- 172.21.15.90:0/1094613857 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c08009550 con 0x7f9c14106f60 2024-02-21T23:41:25.275 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.389+0000 7f9c1259c700 1 --2- 172.21.15.90:0/1094613857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c14106f60 0x7f9c1419a2c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9c08003e60 tx=0x7f9c08011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:25.276 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.389+0000 7f9c037fe700 1 -- 172.21.15.90:0/1094613857 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9c080038b0 con 0x7f9c14106f60 2024-02-21T23:41:25.276 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.389+0000 7f9c037fe700 1 -- 172.21.15.90:0/1094613857 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9c08003a10 con 0x7f9c14106f60 2024-02-21T23:41:25.276 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.389+0000 7f9c18e40700 1 -- 172.21.15.90:0/1094613857 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c1419aa00 con 0x7f9c14106f60 2024-02-21T23:41:25.276 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.390+0000 7f9c037fe700 1 -- 172.21.15.90:0/1094613857 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9c08019530 con 0x7f9c14106f60 2024-02-21T23:41:25.276 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.390+0000 7f9c18e40700 1 -- 172.21.15.90:0/1094613857 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c1419aea0 con 0x7f9c14106f60 2024-02-21T23:41:25.276 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.390+0000 7f9c18e40700 1 -- 172.21.15.90:0/1094613857 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c1410a790 con 0x7f9c14106f60 2024-02-21T23:41:25.276 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.391+0000 7f9c037fe700 1 -- 172.21.15.90:0/1094613857 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) v1 ==== 45537+0+0 (secure 0 0 0) 0x7f9c0801a980 con 0x7f9c14106f60 2024-02-21T23:41:25.276 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.391+0000 7f9c037fe700 1 --2- 172.21.15.90:0/1094613857 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f9bfc0386b0 0x7f9bfc03ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:25.276 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.391+0000 7f9c037fe700 1 -- 172.21.15.90:0/1094613857 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f9c0804c880 con 0x7f9c14106f60 2024-02-21T23:41:25.276 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.391+0000 7f9c11d9b700 1 --2- 172.21.15.90:0/1094613857 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f9bfc0386b0 0x7f9bfc03ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:25.276 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.391+0000 7f9c11d9b700 1 --2- 172.21.15.90:0/1094613857 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f9bfc0386b0 0x7f9bfc03ab70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9c04006fd0 tx=0x7f9c04006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:25.277 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.395+0000 7f9c037fe700 1 -- 172.21.15.90:0/1094613857 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f9c08003b80 con 0x7f9c14106f60 2024-02-21T23:41:25.277 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.593+0000 7f9c18e40700 1 -- 172.21.15.90:0/1094613857 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f9c1410a980 con 0x7f9c14106f60 2024-02-21T23:41:25.277 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.596+0000 7f9c037fe700 1 -- 172.21.15.90:0/1094613857 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f9c0804c1f0 con 0x7f9c14106f60 2024-02-21T23:41:25.277 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.598+0000 7f9c18e40700 1 -- 172.21.15.90:0/1094613857 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f9bfc0386b0 msgr2=0x7f9bfc03ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:25.277 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.598+0000 7f9c18e40700 1 --2- 172.21.15.90:0/1094613857 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f9bfc0386b0 0x7f9bfc03ab70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9c04006fd0 tx=0x7f9c04006e40 comp rx=0 tx=0).stop 2024-02-21T23:41:25.277 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.598+0000 7f9c18e40700 1 -- 172.21.15.90:0/1094613857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c14106f60 msgr2=0x7f9c1419a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:25.277 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.598+0000 7f9c18e40700 1 --2- 172.21.15.90:0/1094613857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c14106f60 0x7f9c1419a2c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9c08003e60 tx=0x7f9c08011a30 comp rx=0 tx=0).stop 2024-02-21T23:41:25.277 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.598+0000 7f9c18e40700 1 -- 172.21.15.90:0/1094613857 shutdown_connections 2024-02-21T23:41:25.277 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.598+0000 7f9c18e40700 1 --2- 172.21.15.90:0/1094613857 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f9bfc0386b0 0x7f9bfc03ab70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:25.277 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.598+0000 7f9c18e40700 1 --2- 172.21.15.90:0/1094613857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c14106f60 0x7f9c1419a2c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:25.277 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.598+0000 7f9c18e40700 1 -- 172.21.15.90:0/1094613857 >> 172.21.15.90:0/1094613857 conn(0x7f9c14100970 msgr2=0x7f9c14101500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:25.278 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.598+0000 7f9c18e40700 1 -- 172.21.15.90:0/1094613857 shutdown_connections 2024-02-21T23:41:25.278 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:24.598+0000 7f9c18e40700 1 -- 172.21.15.90:0/1094613857 wait complete. 2024-02-21T23:41:25.278 INFO:teuthology.orchestra.run.smithi090.stdout:Enabling the dashboard module... 2024-02-21T23:41:25.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:25 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/1094613857' entity='client.admin' 2024-02-21T23:41:26.845 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:26 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/3950617968' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-21T23:41:26.845 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:26 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: ignoring --setuser ceph since I am not root 2024-02-21T23:41:26.845 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:26 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: ignoring --setgroup ceph since I am not root 2024-02-21T23:41:26.985 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.633+0000 7f8fadabb700 1 Processor -- start 2024-02-21T23:41:26.991 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.633+0000 7f8fadabb700 1 -- start start 2024-02-21T23:41:26.991 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.633+0000 7f8fadabb700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fa8106490 0x7f8fa81068b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:26.991 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.633+0000 7f8fadabb700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa8106e80 con 0x7f8fa8106490 2024-02-21T23:41:26.991 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.634+0000 7f8fa77fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fa8106490 0x7f8fa81068b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:26.991 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.634+0000 7f8fa77fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fa8106490 0x7f8fa81068b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57380/0 (socket says 172.21.15.90:57380) 2024-02-21T23:41:26.991 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.634+0000 7f8fa77fe700 1 -- 172.21.15.90:0/2037811704 learned_addr learned my addr 172.21.15.90:0/2037811704 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:26.991 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.634+0000 7f8fa77fe700 1 -- 172.21.15.90:0/2037811704 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa8107690 con 0x7f8fa8106490 2024-02-21T23:41:26.991 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.634+0000 7f8fa77fe700 1 --2- 172.21.15.90:0/2037811704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fa8106490 0x7f8fa81068b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8f980098a0 tx=0x7f8f98009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7750d41ad9e8e370 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:26.992 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.634+0000 7f8fa67fc700 1 -- 172.21.15.90:0/2037811704 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8f98004160 con 0x7f8fa8106490 2024-02-21T23:41:26.992 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.634+0000 7f8fa67fc700 1 -- 172.21.15.90:0/2037811704 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f980042c0 con 0x7f8fa8106490 2024-02-21T23:41:26.992 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.634+0000 7f8fa67fc700 1 -- 172.21.15.90:0/2037811704 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8f980119b0 con 0x7f8fa8106490 2024-02-21T23:41:26.992 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.635+0000 7f8fadabb700 1 -- 172.21.15.90:0/2037811704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fa8106490 msgr2=0x7f8fa81068b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:26.992 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.635+0000 7f8fadabb700 1 --2- 172.21.15.90:0/2037811704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fa8106490 0x7f8fa81068b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8f980098a0 tx=0x7f8f98009bb0 comp rx=0 tx=0).stop 2024-02-21T23:41:26.992 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.635+0000 7f8fadabb700 1 -- 172.21.15.90:0/2037811704 shutdown_connections 2024-02-21T23:41:26.992 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.635+0000 7f8fadabb700 1 --2- 172.21.15.90:0/2037811704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fa8106490 0x7f8fa81068b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:26.992 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.635+0000 7f8fadabb700 1 -- 172.21.15.90:0/2037811704 >> 172.21.15.90:0/2037811704 conn(0x7f8fa8101a90 msgr2=0x7f8fa8103eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:26.992 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.635+0000 7f8fadabb700 1 -- 172.21.15.90:0/2037811704 shutdown_connections 2024-02-21T23:41:26.992 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.635+0000 7f8fadabb700 1 -- 172.21.15.90:0/2037811704 wait complete. 2024-02-21T23:41:26.992 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.635+0000 7f8fadabb700 1 Processor -- start 2024-02-21T23:41:26.992 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.635+0000 7f8fadabb700 1 -- start start 2024-02-21T23:41:26.992 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.636+0000 7f8fadabb700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fa8106490 0x7f8fa819a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:26.993 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.636+0000 7f8fadabb700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa819a760 con 0x7f8fa8106490 2024-02-21T23:41:26.993 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.636+0000 7f8fa77fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fa8106490 0x7f8fa819a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:26.993 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.636+0000 7f8fa77fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fa8106490 0x7f8fa819a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57384/0 (socket says 172.21.15.90:57384) 2024-02-21T23:41:26.993 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.636+0000 7f8fa77fe700 1 -- 172.21.15.90:0/3950617968 learned_addr learned my addr 172.21.15.90:0/3950617968 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:26.993 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.636+0000 7f8fa77fe700 1 -- 172.21.15.90:0/3950617968 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f98009550 con 0x7f8fa8106490 2024-02-21T23:41:26.993 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.636+0000 7f8fa77fe700 1 --2- 172.21.15.90:0/3950617968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fa8106490 0x7f8fa819a220 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8f98006af0 tx=0x7f8f98011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:26.993 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.636+0000 7f8fa4ff9700 1 -- 172.21.15.90:0/3950617968 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8f98003a40 con 0x7f8fa8106490 2024-02-21T23:41:26.993 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.636+0000 7f8fa4ff9700 1 -- 172.21.15.90:0/3950617968 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f98003ba0 con 0x7f8fa8106490 2024-02-21T23:41:26.993 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.636+0000 7f8fadabb700 1 -- 172.21.15.90:0/3950617968 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fa819a960 con 0x7f8fa8106490 2024-02-21T23:41:26.993 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.636+0000 7f8fa4ff9700 1 -- 172.21.15.90:0/3950617968 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8f98019690 con 0x7f8fa8106490 2024-02-21T23:41:26.993 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.636+0000 7f8fadabb700 1 -- 172.21.15.90:0/3950617968 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fa819ae00 con 0x7f8fa8106490 2024-02-21T23:41:26.993 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.637+0000 7f8fa4ff9700 1 -- 172.21.15.90:0/3950617968 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) v1 ==== 45537+0+0 (secure 0 0 0) 0x7f8f98019900 con 0x7f8fa8106490 2024-02-21T23:41:26.993 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.637+0000 7f8fadabb700 1 -- 172.21.15.90:0/3950617968 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fa810a8c0 con 0x7f8fa8106490 2024-02-21T23:41:26.993 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.637+0000 7f8fa4ff9700 1 --2- 172.21.15.90:0/3950617968 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f8f94038700 0x7f8f9403abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:26.994 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.637+0000 7f8fa4ff9700 1 -- 172.21.15.90:0/3950617968 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f8f9804c5e0 con 0x7f8fa8106490 2024-02-21T23:41:26.994 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.637+0000 7f8fa6ffd700 1 --2- 172.21.15.90:0/3950617968 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f8f94038700 0x7f8f9403abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:26.994 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.638+0000 7f8fa6ffd700 1 --2- 172.21.15.90:0/3950617968 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f8f94038700 0x7f8f9403abc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8f90006fd0 tx=0x7f8f90006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:26.994 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.642+0000 7f8fa4ff9700 1 -- 172.21.15.90:0/3950617968 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f8f98018350 con 0x7f8fa8106490 2024-02-21T23:41:26.994 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:25.889+0000 7f8fadabb700 1 -- 172.21.15.90:0/3950617968 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f8fa810aae0 con 0x7f8fa8106490 2024-02-21T23:41:26.994 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:26.599+0000 7f8fa4ff9700 1 -- 172.21.15.90:0/3950617968 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mgrmap(e 9) v1 ==== 45550+0+0 (secure 0 0 0) 0x7f8f9802d9a0 con 0x7f8fa8106490 2024-02-21T23:41:26.994 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:26.601+0000 7f8fa4ff9700 1 -- 172.21.15.90:0/3950617968 <== mon.0 v2:172.21.15.90:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f8f9804c340 con 0x7f8fa8106490 2024-02-21T23:41:26.994 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:26.604+0000 7f8fadabb700 1 -- 172.21.15.90:0/3950617968 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f8f94038700 msgr2=0x7f8f9403abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:26.994 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:26.604+0000 7f8fadabb700 1 --2- 172.21.15.90:0/3950617968 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f8f94038700 0x7f8f9403abc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8f90006fd0 tx=0x7f8f90006e40 comp rx=0 tx=0).stop 2024-02-21T23:41:26.994 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:26.604+0000 7f8fadabb700 1 -- 172.21.15.90:0/3950617968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fa8106490 msgr2=0x7f8fa819a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:26.994 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:26.604+0000 7f8fadabb700 1 --2- 172.21.15.90:0/3950617968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fa8106490 0x7f8fa819a220 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8f98006af0 tx=0x7f8f98011a10 comp rx=0 tx=0).stop 2024-02-21T23:41:26.994 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:26.604+0000 7f8fadabb700 1 -- 172.21.15.90:0/3950617968 shutdown_connections 2024-02-21T23:41:26.994 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:26.604+0000 7f8fadabb700 1 --2- 172.21.15.90:0/3950617968 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f8f94038700 0x7f8f9403abc0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:26.995 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:26.604+0000 7f8fadabb700 1 --2- 172.21.15.90:0/3950617968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fa8106490 0x7f8fa819a220 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:26.995 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:26.604+0000 7f8fadabb700 1 -- 172.21.15.90:0/3950617968 >> 172.21.15.90:0/3950617968 conn(0x7f8fa8101a90 msgr2=0x7f8fa81026d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:26.995 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:26.604+0000 7f8fadabb700 1 -- 172.21.15.90:0/3950617968 shutdown_connections 2024-02-21T23:41:26.995 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:26.604+0000 7f8fadabb700 1 -- 172.21.15.90:0/3950617968 wait complete. 2024-02-21T23:41:27.100 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:27 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:27.050+0000 7fd1b1b34200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-21T23:41:27.477 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:27 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:27.150+0000 7fd1b1b34200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-21T23:41:27.478 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:27 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:27.241+0000 7fd1b1b34200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-21T23:41:27.478 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:27 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:27.329+0000 7fd1b1b34200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-21T23:41:27.955 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:27 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/3950617968' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-21T23:41:27.956 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:27 smithi090 ceph-mon[43385]: mgrmap e9: x(active, since 15s) 2024-02-21T23:41:28.227 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:27 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:27.970+0000 7fd1b1b34200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-21T23:41:28.227 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:28 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:28.219+0000 7fd1b1b34200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-21T23:41:28.556 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:28 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:28.305+0000 7fd1b1b34200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-21T23:41:28.557 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:28 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:28.414+0000 7fd1b1b34200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-21T23:41:28.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:28 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/2915574211' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-21T23:41:28.978 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:28 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:28.572+0000 7fd1b1b34200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-21T23:41:28.978 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:28 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:28.703+0000 7fd1b1b34200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-21T23:41:28.978 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:28 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:28.787+0000 7fd1b1b34200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-21T23:41:29.477 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:29 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:29.157+0000 7fd1b1b34200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-21T23:41:29.689 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout { 2024-02-21T23:41:29.689 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-02-21T23:41:29.689 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-21T23:41:29.689 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-02-21T23:41:29.689 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-21T23:41:29.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout } 2024-02-21T23:41:29.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.359+0000 7ff22244d700 1 Processor -- start 2024-02-21T23:41:29.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.359+0000 7ff22244d700 1 -- start start 2024-02-21T23:41:29.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.359+0000 7ff22244d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff21c104da0 0x7ff21c1051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:29.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.359+0000 7ff22244d700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21c079e00 con 0x7ff21c104da0 2024-02-21T23:41:29.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.360+0000 7ff21bfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff21c104da0 0x7ff21c1051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:29.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.360+0000 7ff21bfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff21c104da0 0x7ff21c1051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57404/0 (socket says 172.21.15.90:57404) 2024-02-21T23:41:29.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.360+0000 7ff21bfff700 1 -- 172.21.15.90:0/3532885660 learned_addr learned my addr 172.21.15.90:0/3532885660 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:29.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.360+0000 7ff21bfff700 1 -- 172.21.15.90:0/3532885660 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff21c079f40 con 0x7ff21c104da0 2024-02-21T23:41:29.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.360+0000 7ff21bfff700 1 --2- 172.21.15.90:0/3532885660 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff21c104da0 0x7ff21c1051c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff20c0098a0 tx=0x7ff20c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7edb34d6d7691584 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:29.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.360+0000 7ff21affd700 1 -- 172.21.15.90:0/3532885660 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff20c004160 con 0x7ff21c104da0 2024-02-21T23:41:29.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.360+0000 7ff21affd700 1 -- 172.21.15.90:0/3532885660 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff20c0042c0 con 0x7ff21c104da0 2024-02-21T23:41:29.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.360+0000 7ff21affd700 1 -- 172.21.15.90:0/3532885660 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff20c0119b0 con 0x7ff21c104da0 2024-02-21T23:41:29.691 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.361+0000 7ff22244d700 1 -- 172.21.15.90:0/3532885660 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff21c104da0 msgr2=0x7ff21c1051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:29.691 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.361+0000 7ff22244d700 1 --2- 172.21.15.90:0/3532885660 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff21c104da0 0x7ff21c1051c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff20c0098a0 tx=0x7ff20c009bb0 comp rx=0 tx=0).stop 2024-02-21T23:41:29.691 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.361+0000 7ff22244d700 1 -- 172.21.15.90:0/3532885660 shutdown_connections 2024-02-21T23:41:29.691 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.361+0000 7ff22244d700 1 --2- 172.21.15.90:0/3532885660 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff21c104da0 0x7ff21c1051c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:29.691 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.361+0000 7ff22244d700 1 -- 172.21.15.90:0/3532885660 >> 172.21.15.90:0/3532885660 conn(0x7ff21c100980 msgr2=0x7ff21c102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:29.691 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.361+0000 7ff22244d700 1 -- 172.21.15.90:0/3532885660 shutdown_connections 2024-02-21T23:41:29.691 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.361+0000 7ff22244d700 1 -- 172.21.15.90:0/3532885660 wait complete. 2024-02-21T23:41:29.691 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.362+0000 7ff22244d700 1 Processor -- start 2024-02-21T23:41:29.691 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.362+0000 7ff22244d700 1 -- start start 2024-02-21T23:41:29.692 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.362+0000 7ff22244d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff21c104da0 0x7ff21c19c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:29.692 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.362+0000 7ff22244d700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21c19c930 con 0x7ff21c104da0 2024-02-21T23:41:29.692 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.362+0000 7ff21bfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff21c104da0 0x7ff21c19c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:29.692 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.362+0000 7ff21bfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff21c104da0 0x7ff21c19c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57408/0 (socket says 172.21.15.90:57408) 2024-02-21T23:41:29.692 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.362+0000 7ff21bfff700 1 -- 172.21.15.90:0/2915574211 learned_addr learned my addr 172.21.15.90:0/2915574211 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:29.692 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.362+0000 7ff21bfff700 1 -- 172.21.15.90:0/2915574211 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff20c009550 con 0x7ff21c104da0 2024-02-21T23:41:29.692 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.362+0000 7ff21bfff700 1 --2- 172.21.15.90:0/2915574211 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff21c104da0 0x7ff21c19c3f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff20c003e60 tx=0x7ff20c011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:29.692 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.363+0000 7ff2197fa700 1 -- 172.21.15.90:0/2915574211 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff20c0038b0 con 0x7ff21c104da0 2024-02-21T23:41:29.692 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.363+0000 7ff2197fa700 1 -- 172.21.15.90:0/2915574211 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff20c003a10 con 0x7ff21c104da0 2024-02-21T23:41:29.692 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.363+0000 7ff22244d700 1 -- 172.21.15.90:0/2915574211 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff21c19cb30 con 0x7ff21c104da0 2024-02-21T23:41:29.693 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.363+0000 7ff2197fa700 1 -- 172.21.15.90:0/2915574211 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff20c019530 con 0x7ff21c104da0 2024-02-21T23:41:29.693 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.363+0000 7ff22244d700 1 -- 172.21.15.90:0/2915574211 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff21c19cfd0 con 0x7ff21c104da0 2024-02-21T23:41:29.693 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.363+0000 7ff2197fa700 1 -- 172.21.15.90:0/2915574211 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 9) v1 ==== 45550+0+0 (secure 0 0 0) 0x7ff20c0197f0 con 0x7ff21c104da0 2024-02-21T23:41:29.693 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.364+0000 7ff22244d700 1 -- 172.21.15.90:0/2915574211 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff21c196120 con 0x7ff21c104da0 2024-02-21T23:41:29.693 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.364+0000 7ff2197fa700 1 --2- 172.21.15.90:0/2915574211 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7ff204038700 0x7ff20403abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:29.693 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.364+0000 7ff2197fa700 1 -- 172.21.15.90:0/2915574211 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff20c04d870 con 0x7ff21c104da0 2024-02-21T23:41:29.693 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.364+0000 7ff21b7fe700 1 -- 172.21.15.90:0/2915574211 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7ff204038700 msgr2=0x7ff20403abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/1385551604 2024-02-21T23:41:29.693 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.364+0000 7ff21b7fe700 1 --2- 172.21.15.90:0/2915574211 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7ff204038700 0x7ff20403abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-21T23:41:29.693 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.368+0000 7ff2197fa700 1 -- 172.21.15.90:0/2915574211 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ff20c02b340 con 0x7ff21c104da0 2024-02-21T23:41:29.693 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.565+0000 7ff21b7fe700 1 -- 172.21.15.90:0/2915574211 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7ff204038700 msgr2=0x7ff20403abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/1385551604 2024-02-21T23:41:29.694 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.565+0000 7ff21b7fe700 1 --2- 172.21.15.90:0/2915574211 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7ff204038700 0x7ff20403abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-21T23:41:29.694 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.621+0000 7ff22244d700 1 -- 172.21.15.90:0/2915574211 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7ff21c02cc70 con 0x7ff21c104da0 2024-02-21T23:41:29.694 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.621+0000 7ff2197fa700 1 -- 172.21.15.90:0/2915574211 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7ff20c018b40 con 0x7ff21c104da0 2024-02-21T23:41:29.694 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.623+0000 7ff22244d700 1 -- 172.21.15.90:0/2915574211 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7ff204038700 msgr2=0x7ff20403abc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-21T23:41:29.694 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.623+0000 7ff22244d700 1 --2- 172.21.15.90:0/2915574211 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7ff204038700 0x7ff20403abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:29.694 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.623+0000 7ff22244d700 1 -- 172.21.15.90:0/2915574211 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff21c104da0 msgr2=0x7ff21c19c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:29.694 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.623+0000 7ff22244d700 1 --2- 172.21.15.90:0/2915574211 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff21c104da0 0x7ff21c19c3f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff20c003e60 tx=0x7ff20c011a30 comp rx=0 tx=0).stop 2024-02-21T23:41:29.694 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.623+0000 7ff22244d700 1 -- 172.21.15.90:0/2915574211 shutdown_connections 2024-02-21T23:41:29.694 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.623+0000 7ff22244d700 1 --2- 172.21.15.90:0/2915574211 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7ff204038700 0x7ff20403abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:29.694 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.623+0000 7ff22244d700 1 --2- 172.21.15.90:0/2915574211 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff21c104da0 0x7ff21c19c3f0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:29.694 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.623+0000 7ff22244d700 1 -- 172.21.15.90:0/2915574211 >> 172.21.15.90:0/2915574211 conn(0x7ff21c100980 msgr2=0x7ff21c193260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:29.694 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.623+0000 7ff22244d700 1 -- 172.21.15.90:0/2915574211 shutdown_connections 2024-02-21T23:41:29.694 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:27.623+0000 7ff22244d700 1 -- 172.21.15.90:0/2915574211 wait complete. 2024-02-21T23:41:29.694 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for the mgr to restart... 2024-02-21T23:41:29.695 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for mgr epoch 9... 2024-02-21T23:41:29.790 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:29 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:29.544+0000 7fd1b1b34200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-21T23:41:29.790 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:29 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:29.743+0000 7fd1b1b34200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-21T23:41:30.227 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:29 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:29.850+0000 7fd1b1b34200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-21T23:41:30.977 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:30 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:30.615+0000 7fd1b1b34200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-21T23:41:30.978 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:30 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:30.712+0000 7fd1b1b34200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-21T23:41:30.978 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:30 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:30.859+0000 7fd1b1b34200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-21T23:41:31.727 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:31 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:31.479+0000 7fd1b1b34200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-21T23:41:31.727 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:31 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:31.660+0000 7fd1b1b34200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-21T23:41:32.227 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:31 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:31.854+0000 7fd1b1b34200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-21T23:41:33.049 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:32 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:32.779+0000 7fd1b1b34200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-21T23:41:33.330 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:33 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:33.064+0000 7fd1b1b34200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-21T23:41:33.331 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:33 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:33.153+0000 7fd1b1b34200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-21T23:41:33.731 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:41:33 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:41:33.346+0000 7fd1b1b34200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-21T23:41:33.732 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:33 smithi090 ceph-mon[43385]: Active manager daemon x restarted 2024-02-21T23:41:33.732 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:33 smithi090 ceph-mon[43385]: Activating manager daemon x 2024-02-21T23:41:33.732 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:33 smithi090 ceph-mon[43385]: osdmap e3: 0 total, 0 up, 0 in 2024-02-21T23:41:33.732 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:33 smithi090 ceph-mon[43385]: mgrmap e10: x(active, starting, since 0.00335128s) 2024-02-21T23:41:33.732 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:33 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-21T23:41:33.732 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:33 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-21T23:41:33.732 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:33 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-21T23:41:33.732 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:33 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-21T23:41:33.732 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:33 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-21T23:41:33.732 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:33 smithi090 ceph-mon[43385]: Manager daemon x is now available 2024-02-21T23:41:33.732 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:33 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:41:33.732 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:33 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:41:34.657 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:34 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-21T23:41:34.658 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:34 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-21T23:41:34.658 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:34 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:41:34.658 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:34 smithi090 ceph-mon[43385]: mgrmap e11: x(active, since 1.00581s) 2024-02-21T23:41:34.842 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout { 2024-02-21T23:41:34.842 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-02-21T23:41:34.842 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-21T23:41:34.842 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout } 2024-02-21T23:41:34.842 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.224+0000 7f215b7a0700 1 Processor -- start 2024-02-21T23:41:34.842 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.224+0000 7f215b7a0700 1 -- start start 2024-02-21T23:41:34.842 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.225+0000 7f215b7a0700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2154106f90 0x7f2154109380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:34.842 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.225+0000 7f215b7a0700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2154079f20 con 0x7f2154106f90 2024-02-21T23:41:34.842 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.225+0000 7f215953c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2154106f90 0x7f2154109380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:34.842 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.225+0000 7f215953c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2154106f90 0x7f2154109380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36386/0 (socket says 172.21.15.90:36386) 2024-02-21T23:41:34.842 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.225+0000 7f215953c700 1 -- 172.21.15.90:0/1671982971 learned_addr learned my addr 172.21.15.90:0/1671982971 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:34.843 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.225+0000 7f215953c700 1 -- 172.21.15.90:0/1671982971 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21541098c0 con 0x7f2154106f90 2024-02-21T23:41:34.843 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.225+0000 7f215953c700 1 --2- 172.21.15.90:0/1671982971 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2154106f90 0x7f2154109380 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f21500098a0 tx=0x7f2150009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e169a4f4b7f68587 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:34.843 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.225+0000 7f214bfff700 1 -- 172.21.15.90:0/1671982971 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2150004160 con 0x7f2154106f90 2024-02-21T23:41:34.843 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.225+0000 7f214bfff700 1 -- 172.21.15.90:0/1671982971 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f21500042c0 con 0x7f2154106f90 2024-02-21T23:41:34.843 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.225+0000 7f214bfff700 1 -- 172.21.15.90:0/1671982971 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f21500119b0 con 0x7f2154106f90 2024-02-21T23:41:34.843 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.226+0000 7f215b7a0700 1 -- 172.21.15.90:0/1671982971 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2154106f90 msgr2=0x7f2154109380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:34.843 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.226+0000 7f215b7a0700 1 --2- 172.21.15.90:0/1671982971 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2154106f90 0x7f2154109380 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f21500098a0 tx=0x7f2150009bb0 comp rx=0 tx=0).stop 2024-02-21T23:41:34.843 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.226+0000 7f215b7a0700 1 -- 172.21.15.90:0/1671982971 shutdown_connections 2024-02-21T23:41:34.843 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.226+0000 7f215b7a0700 1 --2- 172.21.15.90:0/1671982971 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2154106f90 0x7f2154109380 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:34.843 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.226+0000 7f215b7a0700 1 -- 172.21.15.90:0/1671982971 >> 172.21.15.90:0/1671982971 conn(0x7f21541009a0 msgr2=0x7f2154102dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:34.843 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.226+0000 7f215b7a0700 1 -- 172.21.15.90:0/1671982971 shutdown_connections 2024-02-21T23:41:34.843 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.226+0000 7f215b7a0700 1 -- 172.21.15.90:0/1671982971 wait complete. 2024-02-21T23:41:34.843 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.227+0000 7f215b7a0700 1 Processor -- start 2024-02-21T23:41:34.843 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.227+0000 7f215b7a0700 1 -- start start 2024-02-21T23:41:34.844 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.227+0000 7f215b7a0700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2154106f90 0x7f21541067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:34.844 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.227+0000 7f215b7a0700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2154106d00 con 0x7f2154106f90 2024-02-21T23:41:34.844 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.227+0000 7f215953c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2154106f90 0x7f21541067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:34.844 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.227+0000 7f215953c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2154106f90 0x7f21541067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36400/0 (socket says 172.21.15.90:36400) 2024-02-21T23:41:34.844 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.227+0000 7f215953c700 1 -- 172.21.15.90:0/4286062225 learned_addr learned my addr 172.21.15.90:0/4286062225 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:34.844 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.227+0000 7f215953c700 1 -- 172.21.15.90:0/4286062225 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2150009550 con 0x7f2154106f90 2024-02-21T23:41:34.844 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.227+0000 7f215953c700 1 --2- 172.21.15.90:0/4286062225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2154106f90 0x7f21541067c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2150003e60 tx=0x7f2150011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:34.844 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.228+0000 7f214a7fc700 1 -- 172.21.15.90:0/4286062225 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f21500038b0 con 0x7f2154106f90 2024-02-21T23:41:34.844 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.228+0000 7f215b7a0700 1 -- 172.21.15.90:0/4286062225 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2154104f40 con 0x7f2154106f90 2024-02-21T23:41:34.844 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.228+0000 7f214a7fc700 1 -- 172.21.15.90:0/4286062225 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2150003a10 con 0x7f2154106f90 2024-02-21T23:41:34.844 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.228+0000 7f214a7fc700 1 -- 172.21.15.90:0/4286062225 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2150019550 con 0x7f2154106f90 2024-02-21T23:41:34.844 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.228+0000 7f215b7a0700 1 -- 172.21.15.90:0/4286062225 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2154105440 con 0x7f2154106f90 2024-02-21T23:41:34.844 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.229+0000 7f214a7fc700 1 -- 172.21.15.90:0/4286062225 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 9) v1 ==== 45550+0+0 (secure 0 0 0) 0x7f2150019810 con 0x7f2154106f90 2024-02-21T23:41:34.845 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.229+0000 7f214a7fc700 1 --2- 172.21.15.90:0/4286062225 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f21400386e0 0x7f214003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:34.845 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.229+0000 7f214a7fc700 1 -- 172.21.15.90:0/4286062225 --> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f214003b2b0 con 0x7f21400386e0 2024-02-21T23:41:34.845 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.229+0000 7f214a7fc700 1 -- 172.21.15.90:0/4286062225 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f215004c870 con 0x7f2154106f90 2024-02-21T23:41:34.845 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.229+0000 7f2158d3b700 1 -- 172.21.15.90:0/4286062225 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f21400386e0 msgr2=0x7f214003aba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/1385551604 2024-02-21T23:41:34.845 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.229+0000 7f2158d3b700 1 --2- 172.21.15.90:0/4286062225 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f21400386e0 0x7f214003aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-21T23:41:34.845 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.430+0000 7f2158d3b700 1 -- 172.21.15.90:0/4286062225 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f21400386e0 msgr2=0x7f214003aba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/1385551604 2024-02-21T23:41:34.845 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.430+0000 7f2158d3b700 1 --2- 172.21.15.90:0/4286062225 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f21400386e0 0x7f214003aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-21T23:41:34.845 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.831+0000 7f2158d3b700 1 -- 172.21.15.90:0/4286062225 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f21400386e0 msgr2=0x7f214003aba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/1385551604 2024-02-21T23:41:34.845 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:30.831+0000 7f2158d3b700 1 --2- 172.21.15.90:0/4286062225 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f21400386e0 0x7f214003aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-02-21T23:41:34.845 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:31.632+0000 7f2158d3b700 1 -- 172.21.15.90:0/4286062225 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f21400386e0 msgr2=0x7f214003aba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/1385551604 2024-02-21T23:41:34.845 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:31.632+0000 7f2158d3b700 1 --2- 172.21.15.90:0/4286062225 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f21400386e0 0x7f214003aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-02-21T23:41:34.845 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:33.233+0000 7f2158d3b700 1 -- 172.21.15.90:0/4286062225 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f21400386e0 msgr2=0x7f214003aba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/1385551604 2024-02-21T23:41:34.845 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:33.233+0000 7f2158d3b700 1 --2- 172.21.15.90:0/4286062225 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f21400386e0 0x7f214003aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-02-21T23:41:34.845 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:33.361+0000 7f214a7fc700 1 -- 172.21.15.90:0/4286062225 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mgrmap(e 10) v1 ==== 45317+0+0 (secure 0 0 0) 0x7f215002d960 con 0x7f2154106f90 2024-02-21T23:41:34.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:33.361+0000 7f214a7fc700 1 -- 172.21.15.90:0/4286062225 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f21400386e0 msgr2=0x7f214003aba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-21T23:41:34.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:33.361+0000 7f214a7fc700 1 --2- 172.21.15.90:0/4286062225 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f21400386e0 0x7f214003aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:34.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:34.362+0000 7f214a7fc700 1 -- 172.21.15.90:0/4286062225 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mgrmap(e 11) v1 ==== 45444+0+0 (secure 0 0 0) 0x7f2150016890 con 0x7f2154106f90 2024-02-21T23:41:34.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:34.362+0000 7f214a7fc700 1 --2- 172.21.15.90:0/4286062225 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f214003bff0 0x7f214003e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:34.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:34.363+0000 7f214a7fc700 1 -- 172.21.15.90:0/4286062225 --> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f214003b2b0 con 0x7f214003bff0 2024-02-21T23:41:34.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:34.363+0000 7f2158d3b700 1 --2- 172.21.15.90:0/4286062225 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f214003bff0 0x7f214003e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:34.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:34.363+0000 7f2158d3b700 1 --2- 172.21.15.90:0/4286062225 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f214003bff0 0x7f214003e3e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2144003a10 tx=0x7f21440092b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:34.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:34.364+0000 7f214a7fc700 1 -- 172.21.15.90:0/4286062225 <== mgr.14150 v2:172.21.15.90:6800/3697394685 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f214003b2b0 con 0x7f214003bff0 2024-02-21T23:41:34.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:34.370+0000 7f215b7a0700 1 -- 172.21.15.90:0/4286062225 --> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f21540623a0 con 0x7f214003bff0 2024-02-21T23:41:34.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:34.371+0000 7f214a7fc700 1 -- 172.21.15.90:0/4286062225 <== mgr.14150 v2:172.21.15.90:6800/3697394685 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f21540623a0 con 0x7f214003bff0 2024-02-21T23:41:34.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:34.371+0000 7f215b7a0700 1 -- 172.21.15.90:0/4286062225 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f214003bff0 msgr2=0x7f214003e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:34.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:34.371+0000 7f215b7a0700 1 --2- 172.21.15.90:0/4286062225 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f214003bff0 0x7f214003e3e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2144003a10 tx=0x7f21440092b0 comp rx=0 tx=0).stop 2024-02-21T23:41:34.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:34.371+0000 7f215b7a0700 1 -- 172.21.15.90:0/4286062225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2154106f90 msgr2=0x7f21541067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:34.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:34.371+0000 7f215b7a0700 1 --2- 172.21.15.90:0/4286062225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2154106f90 0x7f21541067c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2150003e60 tx=0x7f2150011a30 comp rx=0 tx=0).stop 2024-02-21T23:41:34.847 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:34.371+0000 7f215b7a0700 1 -- 172.21.15.90:0/4286062225 shutdown_connections 2024-02-21T23:41:34.847 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:34.371+0000 7f215b7a0700 1 --2- 172.21.15.90:0/4286062225 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f214003bff0 0x7f214003e3e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:34.847 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:34.371+0000 7f215b7a0700 1 --2- 172.21.15.90:0/4286062225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2154106f90 0x7f21541067c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:34.847 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:34.371+0000 7f215b7a0700 1 --2- 172.21.15.90:0/4286062225 >> [v2:172.21.15.90:6800/1385551604,v1:172.21.15.90:6801/1385551604] conn(0x7f21400386e0 0x7f214003aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:34.847 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:34.371+0000 7f215b7a0700 1 -- 172.21.15.90:0/4286062225 >> 172.21.15.90:0/4286062225 conn(0x7f21541009a0 msgr2=0x7f21541015a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:34.847 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:34.371+0000 7f215b7a0700 1 -- 172.21.15.90:0/4286062225 shutdown_connections 2024-02-21T23:41:34.847 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:34.371+0000 7f215b7a0700 1 -- 172.21.15.90:0/4286062225 wait complete. 2024-02-21T23:41:34.847 INFO:teuthology.orchestra.run.smithi090.stdout:mgr epoch 9 is available 2024-02-21T23:41:34.847 INFO:teuthology.orchestra.run.smithi090.stdout:Generating a dashboard self-signed certificate... 2024-02-21T23:41:35.664 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:35 smithi090 ceph-mon[43385]: [21/Feb/2024:23:41:34] ENGINE Bus STARTING 2024-02-21T23:41:35.664 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:35 smithi090 ceph-mon[43385]: [21/Feb/2024:23:41:34] ENGINE Serving on https://172.21.15.90:7150 2024-02-21T23:41:35.664 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:35 smithi090 ceph-mon[43385]: [21/Feb/2024:23:41:34] ENGINE Serving on http://172.21.15.90:8765 2024-02-21T23:41:35.664 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:35 smithi090 ceph-mon[43385]: [21/Feb/2024:23:41:34] ENGINE Bus STARTED 2024-02-21T23:41:35.664 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:35 smithi090 ceph-mon[43385]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-21T23:41:35.664 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:35 smithi090 ceph-mon[43385]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-21T23:41:35.664 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:35 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:41:36.255 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-02-21T23:41:36.256 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.209+0000 7f50ccd57700 1 Processor -- start 2024-02-21T23:41:36.256 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.210+0000 7f50ccd57700 1 -- start start 2024-02-21T23:41:36.256 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.210+0000 7f50ccd57700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50c8107c20 0x7f50c8108040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:36.256 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.210+0000 7f50ccd57700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c8108610 con 0x7f50c8107c20 2024-02-21T23:41:36.256 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.210+0000 7f50c659c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50c8107c20 0x7f50c8108040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:36.256 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.210+0000 7f50c659c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50c8107c20 0x7f50c8108040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36472/0 (socket says 172.21.15.90:36472) 2024-02-21T23:41:36.256 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.210+0000 7f50c659c700 1 -- 172.21.15.90:0/1197907750 learned_addr learned my addr 172.21.15.90:0/1197907750 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:36.256 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.210+0000 7f50c659c700 1 -- 172.21.15.90:0/1197907750 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c81044e0 con 0x7f50c8107c20 2024-02-21T23:41:36.256 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.211+0000 7f50c659c700 1 --2- 172.21.15.90:0/1197907750 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50c8107c20 0x7f50c8108040 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f50b80098a0 tx=0x7f50b8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a026e819340693b1 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:36.256 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.211+0000 7f50c559a700 1 -- 172.21.15.90:0/1197907750 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f50b8004160 con 0x7f50c8107c20 2024-02-21T23:41:36.259 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.211+0000 7f50c559a700 1 -- 172.21.15.90:0/1197907750 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f50b80042c0 con 0x7f50c8107c20 2024-02-21T23:41:36.259 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.211+0000 7f50c559a700 1 -- 172.21.15.90:0/1197907750 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f50b80119b0 con 0x7f50c8107c20 2024-02-21T23:41:36.260 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.211+0000 7f50ccd57700 1 -- 172.21.15.90:0/1197907750 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50c8107c20 msgr2=0x7f50c8108040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:36.260 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.211+0000 7f50ccd57700 1 --2- 172.21.15.90:0/1197907750 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50c8107c20 0x7f50c8108040 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f50b80098a0 tx=0x7f50b8009bb0 comp rx=0 tx=0).stop 2024-02-21T23:41:36.260 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.211+0000 7f50ccd57700 1 -- 172.21.15.90:0/1197907750 shutdown_connections 2024-02-21T23:41:36.260 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.211+0000 7f50ccd57700 1 --2- 172.21.15.90:0/1197907750 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50c8107c20 0x7f50c8108040 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:36.260 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.211+0000 7f50ccd57700 1 -- 172.21.15.90:0/1197907750 >> 172.21.15.90:0/1197907750 conn(0x7f50c81000a0 msgr2=0x7f50c8102500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:36.260 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.211+0000 7f50ccd57700 1 -- 172.21.15.90:0/1197907750 shutdown_connections 2024-02-21T23:41:36.260 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.212+0000 7f50ccd57700 1 -- 172.21.15.90:0/1197907750 wait complete. 2024-02-21T23:41:36.260 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.212+0000 7f50ccd57700 1 Processor -- start 2024-02-21T23:41:36.260 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.212+0000 7f50ccd57700 1 -- start start 2024-02-21T23:41:36.260 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.212+0000 7f50ccd57700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50c8107c20 0x7f50c819a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:36.260 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.212+0000 7f50ccd57700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c819a860 con 0x7f50c8107c20 2024-02-21T23:41:36.260 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.213+0000 7f50c659c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50c8107c20 0x7f50c819a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:36.260 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.213+0000 7f50c659c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50c8107c20 0x7f50c819a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36486/0 (socket says 172.21.15.90:36486) 2024-02-21T23:41:36.261 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.213+0000 7f50c659c700 1 -- 172.21.15.90:0/4005154467 learned_addr learned my addr 172.21.15.90:0/4005154467 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:36.261 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.213+0000 7f50c659c700 1 -- 172.21.15.90:0/4005154467 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b8009550 con 0x7f50c8107c20 2024-02-21T23:41:36.261 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.213+0000 7f50c659c700 1 --2- 172.21.15.90:0/4005154467 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50c8107c20 0x7f50c819a320 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f50b8011980 tx=0x7f50b8011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:36.261 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.213+0000 7f50b77fe700 1 -- 172.21.15.90:0/4005154467 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f50b8003af0 con 0x7f50c8107c20 2024-02-21T23:41:36.261 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.213+0000 7f50ccd57700 1 -- 172.21.15.90:0/4005154467 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50c819aa60 con 0x7f50c8107c20 2024-02-21T23:41:36.261 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.213+0000 7f50b77fe700 1 -- 172.21.15.90:0/4005154467 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f50b8003c50 con 0x7f50c8107c20 2024-02-21T23:41:36.261 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.213+0000 7f50b77fe700 1 -- 172.21.15.90:0/4005154467 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f50b8019760 con 0x7f50c8107c20 2024-02-21T23:41:36.261 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.213+0000 7f50ccd57700 1 -- 172.21.15.90:0/4005154467 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50c819af00 con 0x7f50c8107c20 2024-02-21T23:41:36.261 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.214+0000 7f50b77fe700 1 -- 172.21.15.90:0/4005154467 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 11) v1 ==== 45444+0+0 (secure 0 0 0) 0x7f50b80199a0 con 0x7f50c8107c20 2024-02-21T23:41:36.261 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.214+0000 7f50ccd57700 1 -- 172.21.15.90:0/4005154467 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50c810aa10 con 0x7f50c8107c20 2024-02-21T23:41:36.261 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.214+0000 7f50b77fe700 1 --2- 172.21.15.90:0/4005154467 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f50b0038620 0x7f50b003aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:36.261 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.214+0000 7f50b77fe700 1 -- 172.21.15.90:0/4005154467 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f50b804c910 con 0x7f50c8107c20 2024-02-21T23:41:36.261 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.214+0000 7f50c5d9b700 1 --2- 172.21.15.90:0/4005154467 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f50b0038620 0x7f50b003aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:36.261 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.215+0000 7f50c5d9b700 1 --2- 172.21.15.90:0/4005154467 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f50b0038620 0x7f50b003aae0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f50bc006fd0 tx=0x7f50bc006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:36.262 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.219+0000 7f50b77fe700 1 -- 172.21.15.90:0/4005154467 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f50b8018db0 con 0x7f50c8107c20 2024-02-21T23:41:36.262 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.439+0000 7f50ccd57700 1 -- 172.21.15.90:0/4005154467 --> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f50c81013b0 con 0x7f50b0038620 2024-02-21T23:41:36.262 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.652+0000 7f50b77fe700 1 -- 172.21.15.90:0/4005154467 <== mgr.14150 v2:172.21.15.90:6800/3697394685 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f50c81013b0 con 0x7f50b0038620 2024-02-21T23:41:36.262 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.654+0000 7f50ccd57700 1 -- 172.21.15.90:0/4005154467 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f50b0038620 msgr2=0x7f50b003aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:36.262 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.654+0000 7f50ccd57700 1 --2- 172.21.15.90:0/4005154467 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f50b0038620 0x7f50b003aae0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f50bc006fd0 tx=0x7f50bc006e40 comp rx=0 tx=0).stop 2024-02-21T23:41:36.262 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.654+0000 7f50ccd57700 1 -- 172.21.15.90:0/4005154467 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50c8107c20 msgr2=0x7f50c819a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:36.262 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.654+0000 7f50ccd57700 1 --2- 172.21.15.90:0/4005154467 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50c8107c20 0x7f50c819a320 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f50b8011980 tx=0x7f50b8011a60 comp rx=0 tx=0).stop 2024-02-21T23:41:36.262 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.654+0000 7f50ccd57700 1 -- 172.21.15.90:0/4005154467 shutdown_connections 2024-02-21T23:41:36.262 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.654+0000 7f50ccd57700 1 --2- 172.21.15.90:0/4005154467 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f50b0038620 0x7f50b003aae0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:36.262 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.654+0000 7f50ccd57700 1 --2- 172.21.15.90:0/4005154467 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50c8107c20 0x7f50c819a320 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:36.262 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.654+0000 7f50ccd57700 1 -- 172.21.15.90:0/4005154467 >> 172.21.15.90:0/4005154467 conn(0x7f50c81000a0 msgr2=0x7f50c8100ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:36.262 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.654+0000 7f50ccd57700 1 -- 172.21.15.90:0/4005154467 shutdown_connections 2024-02-21T23:41:36.262 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:35.654+0000 7f50ccd57700 1 -- 172.21.15.90:0/4005154467 wait complete. 2024-02-21T23:41:36.262 INFO:teuthology.orchestra.run.smithi090.stdout:Creating initial admin user... 2024-02-21T23:41:36.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:36 smithi090 ceph-mon[43385]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:41:36.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:36 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:41:36.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:36 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:41:36.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:36 smithi090 ceph-mon[43385]: mgrmap e12: x(active, since 2s) 2024-02-21T23:41:37.439 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$f2laVXr7VSdepmgYgdFNkuFdiuLuu5TASQSH4QYIejZIeOn50NC8K", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1708558897, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-21T23:41:37.439 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.618+0000 7fcd32882700 1 Processor -- start 2024-02-21T23:41:37.440 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.618+0000 7fcd32882700 1 -- start start 2024-02-21T23:41:37.440 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.618+0000 7fcd32882700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcd2c104d70 0x7fcd2c105190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:37.440 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.618+0000 7fcd32882700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2c079f20 con 0x7fcd2c104d70 2024-02-21T23:41:37.440 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.619+0000 7fcd2bfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcd2c104d70 0x7fcd2c105190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:37.440 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.619+0000 7fcd2bfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcd2c104d70 0x7fcd2c105190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36488/0 (socket says 172.21.15.90:36488) 2024-02-21T23:41:37.440 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.619+0000 7fcd2bfff700 1 -- 172.21.15.90:0/669850986 learned_addr learned my addr 172.21.15.90:0/669850986 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:37.440 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.619+0000 7fcd2bfff700 1 -- 172.21.15.90:0/669850986 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd2c1056d0 con 0x7fcd2c104d70 2024-02-21T23:41:37.440 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.619+0000 7fcd2bfff700 1 --2- 172.21.15.90:0/669850986 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcd2c104d70 0x7fcd2c105190 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcd140098a0 tx=0x7fcd14009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19e4d47694de69f7 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:37.440 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.619+0000 7fcd2affd700 1 -- 172.21.15.90:0/669850986 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd14004160 con 0x7fcd2c104d70 2024-02-21T23:41:37.440 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.619+0000 7fcd2affd700 1 -- 172.21.15.90:0/669850986 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcd140042c0 con 0x7fcd2c104d70 2024-02-21T23:41:37.440 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.619+0000 7fcd2affd700 1 -- 172.21.15.90:0/669850986 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd140119b0 con 0x7fcd2c104d70 2024-02-21T23:41:37.440 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.620+0000 7fcd32882700 1 -- 172.21.15.90:0/669850986 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcd2c104d70 msgr2=0x7fcd2c105190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:37.440 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.620+0000 7fcd32882700 1 --2- 172.21.15.90:0/669850986 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcd2c104d70 0x7fcd2c105190 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcd140098a0 tx=0x7fcd14009bb0 comp rx=0 tx=0).stop 2024-02-21T23:41:37.440 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.620+0000 7fcd32882700 1 -- 172.21.15.90:0/669850986 shutdown_connections 2024-02-21T23:41:37.441 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.620+0000 7fcd32882700 1 --2- 172.21.15.90:0/669850986 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcd2c104d70 0x7fcd2c105190 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:37.441 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.620+0000 7fcd32882700 1 -- 172.21.15.90:0/669850986 >> 172.21.15.90:0/669850986 conn(0x7fcd2c100970 msgr2=0x7fcd2c102d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:37.441 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.620+0000 7fcd32882700 1 -- 172.21.15.90:0/669850986 shutdown_connections 2024-02-21T23:41:37.441 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.620+0000 7fcd32882700 1 -- 172.21.15.90:0/669850986 wait complete. 2024-02-21T23:41:37.441 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.620+0000 7fcd32882700 1 Processor -- start 2024-02-21T23:41:37.441 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.620+0000 7fcd32882700 1 -- start start 2024-02-21T23:41:37.441 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.621+0000 7fcd32882700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcd2c104d70 0x7fcd2c19c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:37.441 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.621+0000 7fcd32882700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2c19caa0 con 0x7fcd2c104d70 2024-02-21T23:41:37.441 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.621+0000 7fcd2bfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcd2c104d70 0x7fcd2c19c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:37.441 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.621+0000 7fcd2bfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcd2c104d70 0x7fcd2c19c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36494/0 (socket says 172.21.15.90:36494) 2024-02-21T23:41:37.441 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.621+0000 7fcd2bfff700 1 -- 172.21.15.90:0/1262677547 learned_addr learned my addr 172.21.15.90:0/1262677547 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:37.441 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.621+0000 7fcd2bfff700 1 -- 172.21.15.90:0/1262677547 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd14009550 con 0x7fcd2c104d70 2024-02-21T23:41:37.441 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.621+0000 7fcd2bfff700 1 --2- 172.21.15.90:0/1262677547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcd2c104d70 0x7fcd2c19c560 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcd14003e60 tx=0x7fcd14011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:37.442 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.621+0000 7fcd297fa700 1 -- 172.21.15.90:0/1262677547 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd140038b0 con 0x7fcd2c104d70 2024-02-21T23:41:37.442 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.621+0000 7fcd32882700 1 -- 172.21.15.90:0/1262677547 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd2c19cca0 con 0x7fcd2c104d70 2024-02-21T23:41:37.442 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.621+0000 7fcd297fa700 1 -- 172.21.15.90:0/1262677547 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcd14003a10 con 0x7fcd2c104d70 2024-02-21T23:41:37.442 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.621+0000 7fcd297fa700 1 -- 172.21.15.90:0/1262677547 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd14019530 con 0x7fcd2c104d70 2024-02-21T23:41:37.442 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.621+0000 7fcd32882700 1 -- 172.21.15.90:0/1262677547 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd2c19d140 con 0x7fcd2c104d70 2024-02-21T23:41:37.442 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.622+0000 7fcd297fa700 1 -- 172.21.15.90:0/1262677547 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 12) v1 ==== 45550+0+0 (secure 0 0 0) 0x7fcd140197f0 con 0x7fcd2c104d70 2024-02-21T23:41:37.442 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.622+0000 7fcd32882700 1 -- 172.21.15.90:0/1262677547 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd2c196230 con 0x7fcd2c104d70 2024-02-21T23:41:37.442 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.622+0000 7fcd297fa700 1 --2- 172.21.15.90:0/1262677547 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fcd18038700 0x7fcd1803abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:37.442 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.623+0000 7fcd297fa700 1 -- 172.21.15.90:0/1262677547 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fcd1404c960 con 0x7fcd2c104d70 2024-02-21T23:41:37.442 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.623+0000 7fcd2b7fe700 1 --2- 172.21.15.90:0/1262677547 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fcd18038700 0x7fcd1803abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:37.442 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.623+0000 7fcd2b7fe700 1 --2- 172.21.15.90:0/1262677547 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fcd18038700 0x7fcd1803abc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c006fd0 tx=0x7fcd1c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:37.442 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.627+0000 7fcd297fa700 1 -- 172.21.15.90:0/1262677547 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fcd1402b340 con 0x7fcd2c104d70 2024-02-21T23:41:37.442 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:36.836+0000 7fcd32882700 1 -- 172.21.15.90:0/1262677547 --> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fcd2c0621c0 con 0x7fcd18038700 2024-02-21T23:41:37.442 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.056+0000 7fcd297fa700 1 -- 172.21.15.90:0/1262677547 <== mgr.14150 v2:172.21.15.90:6800/3697394685 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fcd2c0621c0 con 0x7fcd18038700 2024-02-21T23:41:37.443 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.058+0000 7fcd32882700 1 -- 172.21.15.90:0/1262677547 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fcd18038700 msgr2=0x7fcd1803abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:37.443 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.058+0000 7fcd32882700 1 --2- 172.21.15.90:0/1262677547 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fcd18038700 0x7fcd1803abc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c006fd0 tx=0x7fcd1c006e40 comp rx=0 tx=0).stop 2024-02-21T23:41:37.443 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.058+0000 7fcd32882700 1 -- 172.21.15.90:0/1262677547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcd2c104d70 msgr2=0x7fcd2c19c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:37.443 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.058+0000 7fcd32882700 1 --2- 172.21.15.90:0/1262677547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcd2c104d70 0x7fcd2c19c560 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcd14003e60 tx=0x7fcd14011a30 comp rx=0 tx=0).stop 2024-02-21T23:41:37.443 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.058+0000 7fcd32882700 1 -- 172.21.15.90:0/1262677547 shutdown_connections 2024-02-21T23:41:37.443 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.058+0000 7fcd32882700 1 --2- 172.21.15.90:0/1262677547 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fcd18038700 0x7fcd1803abc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:37.443 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.058+0000 7fcd32882700 1 --2- 172.21.15.90:0/1262677547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcd2c104d70 0x7fcd2c19c560 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:37.443 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.058+0000 7fcd32882700 1 -- 172.21.15.90:0/1262677547 >> 172.21.15.90:0/1262677547 conn(0x7fcd2c100970 msgr2=0x7fcd2c193370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:37.443 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.059+0000 7fcd32882700 1 -- 172.21.15.90:0/1262677547 shutdown_connections 2024-02-21T23:41:37.443 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.059+0000 7fcd32882700 1 -- 172.21.15.90:0/1262677547 wait complete. 2024-02-21T23:41:37.443 INFO:teuthology.orchestra.run.smithi090.stdout:Fetching dashboard port number... 2024-02-21T23:41:38.335 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:38 smithi090 ceph-mon[43385]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:41:38.336 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:38 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:41:38.406 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 8443 2024-02-21T23:41:38.406 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.856+0000 7f0bed74b700 1 Processor -- start 2024-02-21T23:41:38.406 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.856+0000 7f0bed74b700 1 -- start start 2024-02-21T23:41:38.406 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.856+0000 7f0bed74b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0be8106fc0 0x7f0be81093b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:38.406 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.856+0000 7f0bed74b700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be8079e40 con 0x7f0be8106fc0 2024-02-21T23:41:38.406 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.857+0000 7f0be6ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0be8106fc0 0x7f0be81093b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:38.406 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.857+0000 7f0be6ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0be8106fc0 0x7f0be81093b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36508/0 (socket says 172.21.15.90:36508) 2024-02-21T23:41:38.406 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.857+0000 7f0be6ffd700 1 -- 172.21.15.90:0/1295269361 learned_addr learned my addr 172.21.15.90:0/1295269361 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:38.406 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.857+0000 7f0be6ffd700 1 -- 172.21.15.90:0/1295269361 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be81098f0 con 0x7f0be8106fc0 2024-02-21T23:41:38.406 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.857+0000 7f0be6ffd700 1 --2- 172.21.15.90:0/1295269361 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0be8106fc0 0x7f0be81093b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0bd80098a0 tx=0x7f0bd8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71aaf50b56d174aa server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:38.406 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.857+0000 7f0be5ffb700 1 -- 172.21.15.90:0/1295269361 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0bd8004160 con 0x7f0be8106fc0 2024-02-21T23:41:38.406 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.857+0000 7f0be5ffb700 1 -- 172.21.15.90:0/1295269361 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0bd80042c0 con 0x7f0be8106fc0 2024-02-21T23:41:38.407 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.857+0000 7f0be5ffb700 1 -- 172.21.15.90:0/1295269361 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0bd80119b0 con 0x7f0be8106fc0 2024-02-21T23:41:38.407 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.858+0000 7f0bed74b700 1 -- 172.21.15.90:0/1295269361 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0be8106fc0 msgr2=0x7f0be81093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:38.407 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.858+0000 7f0bed74b700 1 --2- 172.21.15.90:0/1295269361 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0be8106fc0 0x7f0be81093b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0bd80098a0 tx=0x7f0bd8009bb0 comp rx=0 tx=0).stop 2024-02-21T23:41:38.407 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.858+0000 7f0bed74b700 1 -- 172.21.15.90:0/1295269361 shutdown_connections 2024-02-21T23:41:38.407 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.858+0000 7f0bed74b700 1 --2- 172.21.15.90:0/1295269361 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0be8106fc0 0x7f0be81093b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:38.407 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.858+0000 7f0bed74b700 1 -- 172.21.15.90:0/1295269361 >> 172.21.15.90:0/1295269361 conn(0x7f0be8100a30 msgr2=0x7f0be8102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:38.407 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.858+0000 7f0bed74b700 1 -- 172.21.15.90:0/1295269361 shutdown_connections 2024-02-21T23:41:38.407 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.858+0000 7f0bed74b700 1 -- 172.21.15.90:0/1295269361 wait complete. 2024-02-21T23:41:38.407 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.858+0000 7f0bed74b700 1 Processor -- start 2024-02-21T23:41:38.407 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.859+0000 7f0bed74b700 1 -- start start 2024-02-21T23:41:38.407 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.859+0000 7f0bed74b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0be8106fc0 0x7f0be81067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:38.407 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.859+0000 7f0bed74b700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be8106d30 con 0x7f0be8106fc0 2024-02-21T23:41:38.407 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.859+0000 7f0be6ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0be8106fc0 0x7f0be81067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:38.407 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.859+0000 7f0be6ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0be8106fc0 0x7f0be81067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36524/0 (socket says 172.21.15.90:36524) 2024-02-21T23:41:38.408 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.859+0000 7f0be6ffd700 1 -- 172.21.15.90:0/594763734 learned_addr learned my addr 172.21.15.90:0/594763734 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:38.408 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.859+0000 7f0be6ffd700 1 -- 172.21.15.90:0/594763734 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd8009550 con 0x7f0be8106fc0 2024-02-21T23:41:38.408 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.859+0000 7f0be6ffd700 1 --2- 172.21.15.90:0/594763734 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0be8106fc0 0x7f0be81067f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0bd8011670 tx=0x7f0bd8011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:38.408 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.859+0000 7f0bcffff700 1 -- 172.21.15.90:0/594763734 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0bd8003b20 con 0x7f0be8106fc0 2024-02-21T23:41:38.408 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.859+0000 7f0bcffff700 1 -- 172.21.15.90:0/594763734 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0bd8003c80 con 0x7f0be8106fc0 2024-02-21T23:41:38.408 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.859+0000 7f0bed74b700 1 -- 172.21.15.90:0/594763734 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0be8104f70 con 0x7f0be8106fc0 2024-02-21T23:41:38.408 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.859+0000 7f0bcffff700 1 -- 172.21.15.90:0/594763734 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0bd8019770 con 0x7f0be8106fc0 2024-02-21T23:41:38.408 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.860+0000 7f0bed74b700 1 -- 172.21.15.90:0/594763734 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0be8105410 con 0x7f0be8106fc0 2024-02-21T23:41:38.408 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.860+0000 7f0bcffff700 1 -- 172.21.15.90:0/594763734 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 12) v1 ==== 45550+0+0 (secure 0 0 0) 0x7f0bd8019a30 con 0x7f0be8106fc0 2024-02-21T23:41:38.408 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.860+0000 7f0bed74b700 1 -- 172.21.15.90:0/594763734 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0be810a6d0 con 0x7f0be8106fc0 2024-02-21T23:41:38.408 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.860+0000 7f0bcffff700 1 --2- 172.21.15.90:0/594763734 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f0bd0038750 0x7f0bd003ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:38.408 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.861+0000 7f0bcffff700 1 -- 172.21.15.90:0/594763734 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0bd804ccd0 con 0x7f0be8106fc0 2024-02-21T23:41:38.409 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.861+0000 7f0be67fc700 1 --2- 172.21.15.90:0/594763734 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f0bd0038750 0x7f0bd003ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:38.409 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.861+0000 7f0be67fc700 1 --2- 172.21.15.90:0/594763734 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f0bd0038750 0x7f0bd003ac10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc006fd0 tx=0x7f0bdc006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:38.409 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:37.865+0000 7f0bcffff700 1 -- 172.21.15.90:0/594763734 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f0bd8018800 con 0x7f0be8106fc0 2024-02-21T23:41:38.409 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.058+0000 7f0bed74b700 1 -- 172.21.15.90:0/594763734 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f0be810a8c0 con 0x7f0be8106fc0 2024-02-21T23:41:38.409 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.058+0000 7f0bcffff700 1 -- 172.21.15.90:0/594763734 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f0bd8023e60 con 0x7f0be8106fc0 2024-02-21T23:41:38.409 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.060+0000 7f0bed74b700 1 -- 172.21.15.90:0/594763734 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f0bd0038750 msgr2=0x7f0bd003ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:38.409 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.060+0000 7f0bed74b700 1 --2- 172.21.15.90:0/594763734 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f0bd0038750 0x7f0bd003ac10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc006fd0 tx=0x7f0bdc006e40 comp rx=0 tx=0).stop 2024-02-21T23:41:38.409 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.060+0000 7f0bed74b700 1 -- 172.21.15.90:0/594763734 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0be8106fc0 msgr2=0x7f0be81067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:38.409 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.060+0000 7f0bed74b700 1 --2- 172.21.15.90:0/594763734 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0be8106fc0 0x7f0be81067f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0bd8011670 tx=0x7f0bd8011a30 comp rx=0 tx=0).stop 2024-02-21T23:41:38.409 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.060+0000 7f0bed74b700 1 -- 172.21.15.90:0/594763734 shutdown_connections 2024-02-21T23:41:38.409 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.060+0000 7f0bed74b700 1 --2- 172.21.15.90:0/594763734 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f0bd0038750 0x7f0bd003ac10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:38.409 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.060+0000 7f0bed74b700 1 --2- 172.21.15.90:0/594763734 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0be8106fc0 0x7f0be81067f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:38.409 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.060+0000 7f0bed74b700 1 -- 172.21.15.90:0/594763734 >> 172.21.15.90:0/594763734 conn(0x7f0be8100a30 msgr2=0x7f0be8101540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:38.410 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.060+0000 7f0bed74b700 1 -- 172.21.15.90:0/594763734 shutdown_connections 2024-02-21T23:41:38.410 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.060+0000 7f0bed74b700 1 -- 172.21.15.90:0/594763734 wait complete. 2024-02-21T23:41:38.415 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-21T23:41:38.415 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout disabled 2024-02-21T23:41:38.423 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-21T23:41:38.423 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout inactive 2024-02-21T23:41:38.424 INFO:teuthology.orchestra.run.smithi090.stdout:firewalld.service is not enabled 2024-02-21T23:41:38.424 INFO:teuthology.orchestra.run.smithi090.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-21T23:41:38.425 INFO:teuthology.orchestra.run.smithi090.stdout:Ceph Dashboard is now available at: 2024-02-21T23:41:38.425 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:41:38.425 INFO:teuthology.orchestra.run.smithi090.stdout: URL: https://smithi090:8443/ 2024-02-21T23:41:38.425 INFO:teuthology.orchestra.run.smithi090.stdout: User: admin 2024-02-21T23:41:38.425 INFO:teuthology.orchestra.run.smithi090.stdout: Password: 5yhzhslsd4 2024-02-21T23:41:38.425 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:41:38.426 INFO:teuthology.orchestra.run.smithi090.stdout:Saving cluster configuration to /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/config directory 2024-02-21T23:41:38.426 INFO:teuthology.orchestra.run.smithi090.stdout:Enabling autotune for osd_memory_target 2024-02-21T23:41:39.310 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:39 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/594763734' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-21T23:41:39.347 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.809+0000 7f0acf9a5700 1 Processor -- start 2024-02-21T23:41:39.347 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.809+0000 7f0acf9a5700 1 -- start start 2024-02-21T23:41:39.347 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.809+0000 7f0acf9a5700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ac8106f00 0x7f0ac81092f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:39.347 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.809+0000 7f0acf9a5700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac8109830 con 0x7f0ac8106f00 2024-02-21T23:41:39.347 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.809+0000 7f0acd741700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ac8106f00 0x7f0ac81092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:39.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.809+0000 7f0acd741700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ac8106f00 0x7f0ac81092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34200/0 (socket says 172.21.15.90:34200) 2024-02-21T23:41:39.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.809+0000 7f0acd741700 1 -- 172.21.15.90:0/3471150109 learned_addr learned my addr 172.21.15.90:0/3471150109 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:39.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.809+0000 7f0acd741700 1 -- 172.21.15.90:0/3471150109 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac8109970 con 0x7f0ac8106f00 2024-02-21T23:41:39.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.809+0000 7f0acd741700 1 --2- 172.21.15.90:0/3471150109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ac8106f00 0x7f0ac81092f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0ab80098a0 tx=0x7f0ab8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bcbabcf41cd43d5 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:39.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.810+0000 7f0abffff700 1 -- 172.21.15.90:0/3471150109 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ab8004160 con 0x7f0ac8106f00 2024-02-21T23:41:39.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.810+0000 7f0abffff700 1 -- 172.21.15.90:0/3471150109 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0ab80042c0 con 0x7f0ac8106f00 2024-02-21T23:41:39.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.810+0000 7f0abffff700 1 -- 172.21.15.90:0/3471150109 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ab80119b0 con 0x7f0ac8106f00 2024-02-21T23:41:39.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.810+0000 7f0acf9a5700 1 -- 172.21.15.90:0/3471150109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ac8106f00 msgr2=0x7f0ac81092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:39.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.810+0000 7f0acf9a5700 1 --2- 172.21.15.90:0/3471150109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ac8106f00 0x7f0ac81092f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0ab80098a0 tx=0x7f0ab8009bb0 comp rx=0 tx=0).stop 2024-02-21T23:41:39.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.810+0000 7f0acf9a5700 1 -- 172.21.15.90:0/3471150109 shutdown_connections 2024-02-21T23:41:39.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.810+0000 7f0acf9a5700 1 --2- 172.21.15.90:0/3471150109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ac8106f00 0x7f0ac81092f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:39.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.810+0000 7f0acf9a5700 1 -- 172.21.15.90:0/3471150109 >> 172.21.15.90:0/3471150109 conn(0x7f0ac8100970 msgr2=0x7f0ac8102d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:39.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.810+0000 7f0acf9a5700 1 -- 172.21.15.90:0/3471150109 shutdown_connections 2024-02-21T23:41:39.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.810+0000 7f0acf9a5700 1 -- 172.21.15.90:0/3471150109 wait complete. 2024-02-21T23:41:39.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.811+0000 7f0acf9a5700 1 Processor -- start 2024-02-21T23:41:39.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.811+0000 7f0acf9a5700 1 -- start start 2024-02-21T23:41:39.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.811+0000 7f0acf9a5700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ac8106f00 0x7f0ac819a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:39.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.811+0000 7f0acf9a5700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac819a880 con 0x7f0ac8106f00 2024-02-21T23:41:39.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.812+0000 7f0acd741700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ac8106f00 0x7f0ac819a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:39.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.812+0000 7f0acd741700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ac8106f00 0x7f0ac819a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34214/0 (socket says 172.21.15.90:34214) 2024-02-21T23:41:39.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.812+0000 7f0acd741700 1 -- 172.21.15.90:0/2596377427 learned_addr learned my addr 172.21.15.90:0/2596377427 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:39.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.812+0000 7f0acd741700 1 -- 172.21.15.90:0/2596377427 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab8009550 con 0x7f0ac8106f00 2024-02-21T23:41:39.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.812+0000 7f0acd741700 1 --2- 172.21.15.90:0/2596377427 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ac8106f00 0x7f0ac819a340 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8011980 tx=0x7f0ab8011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:39.352 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.812+0000 7f0abe7fc700 1 -- 172.21.15.90:0/2596377427 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ab8003af0 con 0x7f0ac8106f00 2024-02-21T23:41:39.352 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.812+0000 7f0abe7fc700 1 -- 172.21.15.90:0/2596377427 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0ab8003c50 con 0x7f0ac8106f00 2024-02-21T23:41:39.352 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.812+0000 7f0acf9a5700 1 -- 172.21.15.90:0/2596377427 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ac819aa80 con 0x7f0ac8106f00 2024-02-21T23:41:39.352 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.812+0000 7f0abe7fc700 1 -- 172.21.15.90:0/2596377427 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ab8019760 con 0x7f0ac8106f00 2024-02-21T23:41:39.352 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.812+0000 7f0acf9a5700 1 -- 172.21.15.90:0/2596377427 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ac819af20 con 0x7f0ac8106f00 2024-02-21T23:41:39.352 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.813+0000 7f0abe7fc700 1 -- 172.21.15.90:0/2596377427 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 12) v1 ==== 45550+0+0 (secure 0 0 0) 0x7f0ab801ab10 con 0x7f0ac8106f00 2024-02-21T23:41:39.352 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.813+0000 7f0acf9a5700 1 -- 172.21.15.90:0/2596377427 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ac810a750 con 0x7f0ac8106f00 2024-02-21T23:41:39.352 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.813+0000 7f0abe7fc700 1 --2- 172.21.15.90:0/2596377427 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f0ab40387a0 0x7f0ab403ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:39.352 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.813+0000 7f0abe7fc700 1 -- 172.21.15.90:0/2596377427 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0ab804ca80 con 0x7f0ac8106f00 2024-02-21T23:41:39.352 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.813+0000 7f0accf40700 1 --2- 172.21.15.90:0/2596377427 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f0ab40387a0 0x7f0ab403ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:39.352 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.814+0000 7f0accf40700 1 --2- 172.21.15.90:0/2596377427 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f0ab40387a0 0x7f0ab403ac60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0ac4006fd0 tx=0x7f0ac4006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:39.352 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:38.818+0000 7f0abe7fc700 1 -- 172.21.15.90:0/2596377427 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f0ab801a430 con 0x7f0ac8106f00 2024-02-21T23:41:39.352 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.009+0000 7f0acf9a5700 1 -- 172.21.15.90:0/2596377427 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f0ac810a910 con 0x7f0ac8106f00 2024-02-21T23:41:39.353 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.009+0000 7f0abe7fc700 1 -- 172.21.15.90:0/2596377427 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f0ab802db40 con 0x7f0ac8106f00 2024-02-21T23:41:39.353 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.011+0000 7f0acf9a5700 1 -- 172.21.15.90:0/2596377427 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f0ab40387a0 msgr2=0x7f0ab403ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:39.353 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.011+0000 7f0acf9a5700 1 --2- 172.21.15.90:0/2596377427 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f0ab40387a0 0x7f0ab403ac60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0ac4006fd0 tx=0x7f0ac4006e40 comp rx=0 tx=0).stop 2024-02-21T23:41:39.353 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.011+0000 7f0acf9a5700 1 -- 172.21.15.90:0/2596377427 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ac8106f00 msgr2=0x7f0ac819a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:39.353 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.011+0000 7f0acf9a5700 1 --2- 172.21.15.90:0/2596377427 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ac8106f00 0x7f0ac819a340 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8011980 tx=0x7f0ab8011a60 comp rx=0 tx=0).stop 2024-02-21T23:41:39.353 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.011+0000 7f0acf9a5700 1 -- 172.21.15.90:0/2596377427 shutdown_connections 2024-02-21T23:41:39.353 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.011+0000 7f0acf9a5700 1 --2- 172.21.15.90:0/2596377427 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f0ab40387a0 0x7f0ab403ac60 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:39.353 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.011+0000 7f0acf9a5700 1 --2- 172.21.15.90:0/2596377427 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ac8106f00 0x7f0ac819a340 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:39.353 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.011+0000 7f0acf9a5700 1 -- 172.21.15.90:0/2596377427 >> 172.21.15.90:0/2596377427 conn(0x7f0ac8100970 msgr2=0x7f0ac8101540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:39.353 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.011+0000 7f0acf9a5700 1 -- 172.21.15.90:0/2596377427 shutdown_connections 2024-02-21T23:41:39.353 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.011+0000 7f0acf9a5700 1 -- 172.21.15.90:0/2596377427 wait complete. 2024-02-21T23:41:40.325 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.681+0000 7fdf7ef86700 1 Processor -- start 2024-02-21T23:41:40.332 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.681+0000 7fdf7ef86700 1 -- start start 2024-02-21T23:41:40.332 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.681+0000 7fdf7ef86700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf78104d50 0x7fdf78105170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:40.332 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.681+0000 7fdf7ef86700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf78079f40 con 0x7fdf78104d50 2024-02-21T23:41:40.332 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.681+0000 7fdf7cd22700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf78104d50 0x7fdf78105170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:40.332 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.681+0000 7fdf7cd22700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf78104d50 0x7fdf78105170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34226/0 (socket says 172.21.15.90:34226) 2024-02-21T23:41:40.333 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.681+0000 7fdf7cd22700 1 -- 172.21.15.90:0/1798644213 learned_addr learned my addr 172.21.15.90:0/1798644213 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:40.333 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.682+0000 7fdf7cd22700 1 -- 172.21.15.90:0/1798644213 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf781056b0 con 0x7fdf78104d50 2024-02-21T23:41:40.333 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.682+0000 7fdf7cd22700 1 --2- 172.21.15.90:0/1798644213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf78104d50 0x7fdf78105170 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c0098a0 tx=0x7fdf6c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f33a3ef71d126406 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:40.333 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.682+0000 7fdf777fe700 1 -- 172.21.15.90:0/1798644213 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdf6c004160 con 0x7fdf78104d50 2024-02-21T23:41:40.333 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.682+0000 7fdf777fe700 1 -- 172.21.15.90:0/1798644213 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdf6c0042c0 con 0x7fdf78104d50 2024-02-21T23:41:40.333 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.682+0000 7fdf777fe700 1 -- 172.21.15.90:0/1798644213 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdf6c0119b0 con 0x7fdf78104d50 2024-02-21T23:41:40.333 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.682+0000 7fdf7ef86700 1 -- 172.21.15.90:0/1798644213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf78104d50 msgr2=0x7fdf78105170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:40.333 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.682+0000 7fdf7ef86700 1 --2- 172.21.15.90:0/1798644213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf78104d50 0x7fdf78105170 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c0098a0 tx=0x7fdf6c009bb0 comp rx=0 tx=0).stop 2024-02-21T23:41:40.333 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.683+0000 7fdf7ef86700 1 -- 172.21.15.90:0/1798644213 shutdown_connections 2024-02-21T23:41:40.333 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.683+0000 7fdf7ef86700 1 --2- 172.21.15.90:0/1798644213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf78104d50 0x7fdf78105170 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:40.333 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.683+0000 7fdf7ef86700 1 -- 172.21.15.90:0/1798644213 >> 172.21.15.90:0/1798644213 conn(0x7fdf78100970 msgr2=0x7fdf78102d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:40.333 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.683+0000 7fdf7ef86700 1 -- 172.21.15.90:0/1798644213 shutdown_connections 2024-02-21T23:41:40.334 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.683+0000 7fdf7ef86700 1 -- 172.21.15.90:0/1798644213 wait complete. 2024-02-21T23:41:40.334 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.683+0000 7fdf7ef86700 1 Processor -- start 2024-02-21T23:41:40.334 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.683+0000 7fdf7ef86700 1 -- start start 2024-02-21T23:41:40.334 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.683+0000 7fdf7ef86700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf78104d50 0x7fdf7819c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:40.334 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.683+0000 7fdf7ef86700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7819ca50 con 0x7fdf78104d50 2024-02-21T23:41:40.334 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.683+0000 7fdf7cd22700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf78104d50 0x7fdf7819c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:40.335 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.683+0000 7fdf7cd22700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf78104d50 0x7fdf7819c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34232/0 (socket says 172.21.15.90:34232) 2024-02-21T23:41:40.335 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.683+0000 7fdf7cd22700 1 -- 172.21.15.90:0/1597764648 learned_addr learned my addr 172.21.15.90:0/1597764648 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:40.335 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.684+0000 7fdf7cd22700 1 -- 172.21.15.90:0/1597764648 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf6c009550 con 0x7fdf78104d50 2024-02-21T23:41:40.335 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.684+0000 7fdf7cd22700 1 --2- 172.21.15.90:0/1597764648 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf78104d50 0x7fdf7819c510 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c003e60 tx=0x7fdf6c011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:40.335 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.684+0000 7fdf75ffb700 1 -- 172.21.15.90:0/1597764648 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdf6c0038b0 con 0x7fdf78104d50 2024-02-21T23:41:40.335 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.684+0000 7fdf75ffb700 1 -- 172.21.15.90:0/1597764648 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdf6c003a10 con 0x7fdf78104d50 2024-02-21T23:41:40.335 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.684+0000 7fdf7ef86700 1 -- 172.21.15.90:0/1597764648 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf7819cc50 con 0x7fdf78104d50 2024-02-21T23:41:40.335 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.684+0000 7fdf75ffb700 1 -- 172.21.15.90:0/1597764648 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdf6c019530 con 0x7fdf78104d50 2024-02-21T23:41:40.335 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.684+0000 7fdf7ef86700 1 -- 172.21.15.90:0/1597764648 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf7819d0f0 con 0x7fdf78104d50 2024-02-21T23:41:40.336 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.685+0000 7fdf75ffb700 1 -- 172.21.15.90:0/1597764648 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 12) v1 ==== 45550+0+0 (secure 0 0 0) 0x7fdf6c0197a0 con 0x7fdf78104d50 2024-02-21T23:41:40.336 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.685+0000 7fdf75ffb700 1 --2- 172.21.15.90:0/1597764648 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fdf68038750 0x7fdf6803ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:40.336 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.685+0000 7fdf7ef86700 1 -- 172.21.15.90:0/1597764648 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf78196210 con 0x7fdf78104d50 2024-02-21T23:41:40.336 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.685+0000 7fdf75ffb700 1 -- 172.21.15.90:0/1597764648 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fdf6c04cb40 con 0x7fdf78104d50 2024-02-21T23:41:40.336 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.685+0000 7fdf77fff700 1 --2- 172.21.15.90:0/1597764648 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fdf68038750 0x7fdf6803ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:40.336 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.686+0000 7fdf77fff700 1 --2- 172.21.15.90:0/1597764648 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fdf68038750 0x7fdf6803ac10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fdf64006fd0 tx=0x7fdf64006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:40.336 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.690+0000 7fdf75ffb700 1 -- 172.21.15.90:0/1597764648 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fdf6c019af0 con 0x7fdf78104d50 2024-02-21T23:41:40.336 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.984+0000 7fdf7ef86700 1 -- 172.21.15.90:0/1597764648 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fdf7802cc70 con 0x7fdf78104d50 2024-02-21T23:41:40.337 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.987+0000 7fdf75ffb700 1 -- 172.21.15.90:0/1597764648 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7fdf6c018b40 con 0x7fdf78104d50 2024-02-21T23:41:40.337 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.989+0000 7fdf7ef86700 1 -- 172.21.15.90:0/1597764648 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fdf68038750 msgr2=0x7fdf6803ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:40.337 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.989+0000 7fdf7ef86700 1 --2- 172.21.15.90:0/1597764648 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fdf68038750 0x7fdf6803ac10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fdf64006fd0 tx=0x7fdf64006e40 comp rx=0 tx=0).stop 2024-02-21T23:41:40.337 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.989+0000 7fdf7ef86700 1 -- 172.21.15.90:0/1597764648 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf78104d50 msgr2=0x7fdf7819c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:40.337 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.989+0000 7fdf7ef86700 1 --2- 172.21.15.90:0/1597764648 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf78104d50 0x7fdf7819c510 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c003e60 tx=0x7fdf6c011a30 comp rx=0 tx=0).stop 2024-02-21T23:41:40.337 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.989+0000 7fdf7ef86700 1 -- 172.21.15.90:0/1597764648 shutdown_connections 2024-02-21T23:41:40.337 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.989+0000 7fdf7ef86700 1 --2- 172.21.15.90:0/1597764648 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fdf68038750 0x7fdf6803ac10 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:40.337 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.989+0000 7fdf7ef86700 1 --2- 172.21.15.90:0/1597764648 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf78104d50 0x7fdf7819c510 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:40.337 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.989+0000 7fdf7ef86700 1 -- 172.21.15.90:0/1597764648 >> 172.21.15.90:0/1597764648 conn(0x7fdf78100970 msgr2=0x7fdf78193350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:40.337 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.989+0000 7fdf7ef86700 1 -- 172.21.15.90:0/1597764648 shutdown_connections 2024-02-21T23:41:40.337 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-02-21T23:41:39.989+0000 7fdf7ef86700 1 -- 172.21.15.90:0/1597764648 wait complete. 2024-02-21T23:41:40.338 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-02-21T23:41:40.338 INFO:teuthology.orchestra.run.smithi090.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-02-21T23:41:40.338 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:41:40.338 INFO:teuthology.orchestra.run.smithi090.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-21T23:41:40.338 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:41:40.338 INFO:teuthology.orchestra.run.smithi090.stdout:Or, if you are only running a single cluster on this host: 2024-02-21T23:41:40.338 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:41:40.338 INFO:teuthology.orchestra.run.smithi090.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-02-21T23:41:40.338 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:41:40.338 INFO:teuthology.orchestra.run.smithi090.stdout:Please consider enabling telemetry to help improve Ceph: 2024-02-21T23:41:40.338 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:41:40.338 INFO:teuthology.orchestra.run.smithi090.stdout: ceph telemetry on 2024-02-21T23:41:40.339 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:41:40.339 INFO:teuthology.orchestra.run.smithi090.stdout:For more information see: 2024-02-21T23:41:40.339 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:41:40.339 INFO:teuthology.orchestra.run.smithi090.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-02-21T23:41:40.339 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:41:40.339 INFO:teuthology.orchestra.run.smithi090.stdout:Bootstrap complete. 2024-02-21T23:41:40.365 INFO:tasks.cephadm:Fetching config... 2024-02-21T23:41:40.365 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-21T23:41:40.365 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-21T23:41:40.383 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-21T23:41:40.383 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-21T23:41:40.383 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-21T23:41:40.440 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-21T23:41:40.440 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-21T23:41:40.441 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.a/keyring of=/dev/stdout 2024-02-21T23:41:40.510 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-21T23:41:40.510 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-21T23:41:40.511 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-21T23:41:40.570 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-21T23:41:40.570 DEBUG:teuthology.orchestra.run.smithi090:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDIVpBsFA/ldrPYEoxkgLKW+56KTrEazm9Y3hRa6wwHSIavkZoUJ96kgciX9vSGf6fQithpz5fajRSU5bzciHcL2Xw5UQJedWORPBsHkY4BWmiQnYNO8Uhh+vrAB+Yo/VcsuxiP6rcZ2TCMICNK0K88vcCTQKyrErL/k38e5Py3htLNbMt61M6l+wuSrzeN2U1IWOd+EBfeeSwAVDmeByexUUkOj/4Sg/R2EtK/coJXzvdBZvwpSIhmwy3H0UaGDB73wX96+7tZprLott7k+Qtyc8tOXshEcBAFkOhcdtGNy4+b1mNwsKdccAeInngDUna1FrUbXV55Eh+KJWN99cvsTCru32U3H59q5RHCgy/luT1G6xm04Jwzc0CDJ5fobkTwKwiDJQ6MUtXQP3L4kpyMuDAwotuLmn0LgLaiOwATbao5sylb9PLonVz6OdNvWlzO0jwGbYUV8gMI9yBqUsv0lSTpaOP1oeDRD8HXZt/IRj4qEGNnP6UT5J4V360hKIs= ceph-7ed81218-d112-11ee-95bd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-21T23:41:40.647 INFO:teuthology.orchestra.run.smithi090.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDIVpBsFA/ldrPYEoxkgLKW+56KTrEazm9Y3hRa6wwHSIavkZoUJ96kgciX9vSGf6fQithpz5fajRSU5bzciHcL2Xw5UQJedWORPBsHkY4BWmiQnYNO8Uhh+vrAB+Yo/VcsuxiP6rcZ2TCMICNK0K88vcCTQKyrErL/k38e5Py3htLNbMt61M6l+wuSrzeN2U1IWOd+EBfeeSwAVDmeByexUUkOj/4Sg/R2EtK/coJXzvdBZvwpSIhmwy3H0UaGDB73wX96+7tZprLott7k+Qtyc8tOXshEcBAFkOhcdtGNy4+b1mNwsKdccAeInngDUna1FrUbXV55Eh+KJWN99cvsTCru32U3H59q5RHCgy/luT1G6xm04Jwzc0CDJ5fobkTwKwiDJQ6MUtXQP3L4kpyMuDAwotuLmn0LgLaiOwATbao5sylb9PLonVz6OdNvWlzO0jwGbYUV8gMI9yBqUsv0lSTpaOP1oeDRD8HXZt/IRj4qEGNnP6UT5J4V360hKIs= ceph-7ed81218-d112-11ee-95bd-87774f69a715 2024-02-21T23:41:40.660 DEBUG:teuthology.orchestra.run.smithi176:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDIVpBsFA/ldrPYEoxkgLKW+56KTrEazm9Y3hRa6wwHSIavkZoUJ96kgciX9vSGf6fQithpz5fajRSU5bzciHcL2Xw5UQJedWORPBsHkY4BWmiQnYNO8Uhh+vrAB+Yo/VcsuxiP6rcZ2TCMICNK0K88vcCTQKyrErL/k38e5Py3htLNbMt61M6l+wuSrzeN2U1IWOd+EBfeeSwAVDmeByexUUkOj/4Sg/R2EtK/coJXzvdBZvwpSIhmwy3H0UaGDB73wX96+7tZprLott7k+Qtyc8tOXshEcBAFkOhcdtGNy4+b1mNwsKdccAeInngDUna1FrUbXV55Eh+KJWN99cvsTCru32U3H59q5RHCgy/luT1G6xm04Jwzc0CDJ5fobkTwKwiDJQ6MUtXQP3L4kpyMuDAwotuLmn0LgLaiOwATbao5sylb9PLonVz6OdNvWlzO0jwGbYUV8gMI9yBqUsv0lSTpaOP1oeDRD8HXZt/IRj4qEGNnP6UT5J4V360hKIs= ceph-7ed81218-d112-11ee-95bd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-21T23:41:40.705 INFO:teuthology.orchestra.run.smithi176.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDIVpBsFA/ldrPYEoxkgLKW+56KTrEazm9Y3hRa6wwHSIavkZoUJ96kgciX9vSGf6fQithpz5fajRSU5bzciHcL2Xw5UQJedWORPBsHkY4BWmiQnYNO8Uhh+vrAB+Yo/VcsuxiP6rcZ2TCMICNK0K88vcCTQKyrErL/k38e5Py3htLNbMt61M6l+wuSrzeN2U1IWOd+EBfeeSwAVDmeByexUUkOj/4Sg/R2EtK/coJXzvdBZvwpSIhmwy3H0UaGDB73wX96+7tZprLott7k+Qtyc8tOXshEcBAFkOhcdtGNy4+b1mNwsKdccAeInngDUna1FrUbXV55Eh+KJWN99cvsTCru32U3H59q5RHCgy/luT1G6xm04Jwzc0CDJ5fobkTwKwiDJQ6MUtXQP3L4kpyMuDAwotuLmn0LgLaiOwATbao5sylb9PLonVz6OdNvWlzO0jwGbYUV8gMI9yBqUsv0lSTpaOP1oeDRD8HXZt/IRj4qEGNnP6UT5J4V360hKIs= ceph-7ed81218-d112-11ee-95bd-87774f69a715 2024-02-21T23:41:40.718 DEBUG:teuthology.orchestra.run.smithi178:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDIVpBsFA/ldrPYEoxkgLKW+56KTrEazm9Y3hRa6wwHSIavkZoUJ96kgciX9vSGf6fQithpz5fajRSU5bzciHcL2Xw5UQJedWORPBsHkY4BWmiQnYNO8Uhh+vrAB+Yo/VcsuxiP6rcZ2TCMICNK0K88vcCTQKyrErL/k38e5Py3htLNbMt61M6l+wuSrzeN2U1IWOd+EBfeeSwAVDmeByexUUkOj/4Sg/R2EtK/coJXzvdBZvwpSIhmwy3H0UaGDB73wX96+7tZprLott7k+Qtyc8tOXshEcBAFkOhcdtGNy4+b1mNwsKdccAeInngDUna1FrUbXV55Eh+KJWN99cvsTCru32U3H59q5RHCgy/luT1G6xm04Jwzc0CDJ5fobkTwKwiDJQ6MUtXQP3L4kpyMuDAwotuLmn0LgLaiOwATbao5sylb9PLonVz6OdNvWlzO0jwGbYUV8gMI9yBqUsv0lSTpaOP1oeDRD8HXZt/IRj4qEGNnP6UT5J4V360hKIs= ceph-7ed81218-d112-11ee-95bd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-21T23:41:40.764 INFO:teuthology.orchestra.run.smithi178.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDIVpBsFA/ldrPYEoxkgLKW+56KTrEazm9Y3hRa6wwHSIavkZoUJ96kgciX9vSGf6fQithpz5fajRSU5bzciHcL2Xw5UQJedWORPBsHkY4BWmiQnYNO8Uhh+vrAB+Yo/VcsuxiP6rcZ2TCMICNK0K88vcCTQKyrErL/k38e5Py3htLNbMt61M6l+wuSrzeN2U1IWOd+EBfeeSwAVDmeByexUUkOj/4Sg/R2EtK/coJXzvdBZvwpSIhmwy3H0UaGDB73wX96+7tZprLott7k+Qtyc8tOXshEcBAFkOhcdtGNy4+b1mNwsKdccAeInngDUna1FrUbXV55Eh+KJWN99cvsTCru32U3H59q5RHCgy/luT1G6xm04Jwzc0CDJ5fobkTwKwiDJQ6MUtXQP3L4kpyMuDAwotuLmn0LgLaiOwATbao5sylb9PLonVz6OdNvWlzO0jwGbYUV8gMI9yBqUsv0lSTpaOP1oeDRD8HXZt/IRj4qEGNnP6UT5J4V360hKIs= ceph-7ed81218-d112-11ee-95bd-87774f69a715 2024-02-21T23:41:40.778 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-21T23:41:41.005 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.a/config 2024-02-21T23:41:41.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:40 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/1597764648' entity='client.admin' 2024-02-21T23:41:41.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:40 smithi090 ceph-mon[43385]: mgrmap e13: x(active, since 6s) 2024-02-21T23:41:42.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.163+0000 7fb5a9759700 1 -- 172.21.15.90:0/2419952982 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5a41011e0 msgr2=0x7fb5a41035d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:42.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.163+0000 7fb5a9759700 1 --2- 172.21.15.90:0/2419952982 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5a41011e0 0x7fb5a41035d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb594009910 tx=0x7fb594009c20 comp rx=0 tx=0).stop 2024-02-21T23:41:42.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.163+0000 7fb5a9759700 1 -- 172.21.15.90:0/2419952982 shutdown_connections 2024-02-21T23:41:42.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.163+0000 7fb5a9759700 1 --2- 172.21.15.90:0/2419952982 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5a41011e0 0x7fb5a41035d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:42.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.163+0000 7fb5a9759700 1 -- 172.21.15.90:0/2419952982 >> 172.21.15.90:0/2419952982 conn(0x7fb5a40fab70 msgr2=0x7fb5a40fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:42.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.163+0000 7fb5a9759700 1 -- 172.21.15.90:0/2419952982 shutdown_connections 2024-02-21T23:41:42.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.163+0000 7fb5a9759700 1 -- 172.21.15.90:0/2419952982 wait complete. 2024-02-21T23:41:42.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.163+0000 7fb5a9759700 1 Processor -- start 2024-02-21T23:41:42.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.164+0000 7fb5a9759700 1 -- start start 2024-02-21T23:41:42.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.164+0000 7fb5a9759700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5a41011e0 0x7fb5a4195ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:42.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.164+0000 7fb5a9759700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a4196010 con 0x7fb5a41011e0 2024-02-21T23:41:42.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.164+0000 7fb5a2ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5a41011e0 0x7fb5a4195ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:42.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.164+0000 7fb5a2ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5a41011e0 0x7fb5a4195ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34264/0 (socket says 172.21.15.90:34264) 2024-02-21T23:41:42.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.164+0000 7fb5a2ffd700 1 -- 172.21.15.90:0/2420878741 learned_addr learned my addr 172.21.15.90:0/2420878741 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:42.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.164+0000 7fb5a2ffd700 1 -- 172.21.15.90:0/2420878741 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5940095f0 con 0x7fb5a41011e0 2024-02-21T23:41:42.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.164+0000 7fb5a2ffd700 1 --2- 172.21.15.90:0/2420878741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5a41011e0 0x7fb5a4195ad0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb594005b40 tx=0x7fb5940057e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:42.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.164+0000 7fb58bfff700 1 -- 172.21.15.90:0/2420878741 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb59401d070 con 0x7fb5a41011e0 2024-02-21T23:41:42.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.165+0000 7fb5a9759700 1 -- 172.21.15.90:0/2420878741 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5a4196210 con 0x7fb5a41011e0 2024-02-21T23:41:42.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.165+0000 7fb58bfff700 1 -- 172.21.15.90:0/2420878741 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb59400be20 con 0x7fb5a41011e0 2024-02-21T23:41:42.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.165+0000 7fb58bfff700 1 -- 172.21.15.90:0/2420878741 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb594022d80 con 0x7fb5a41011e0 2024-02-21T23:41:42.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.165+0000 7fb5a9759700 1 -- 172.21.15.90:0/2420878741 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5a41966b0 con 0x7fb5a41011e0 2024-02-21T23:41:42.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.165+0000 7fb58bfff700 1 -- 172.21.15.90:0/2420878741 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) v1 ==== 45593+0+0 (secure 0 0 0) 0x7fb59400f460 con 0x7fb5a41011e0 2024-02-21T23:41:42.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.166+0000 7fb58bfff700 1 --2- 172.21.15.90:0/2420878741 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fb58c0387a0 0x7fb58c03ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:42.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.166+0000 7fb58bfff700 1 -- 172.21.15.90:0/2420878741 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb59404d780 con 0x7fb5a41011e0 2024-02-21T23:41:42.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.166+0000 7fb5a9759700 1 -- 172.21.15.90:0/2420878741 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5a418f3f0 con 0x7fb5a41011e0 2024-02-21T23:41:42.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.166+0000 7fb5a27fc700 1 --2- 172.21.15.90:0/2420878741 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fb58c0387a0 0x7fb58c03ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:42.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.166+0000 7fb5a27fc700 1 --2- 172.21.15.90:0/2420878741 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fb58c0387a0 0x7fb58c03ac60 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb598006fd0 tx=0x7fb598006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:42.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.171+0000 7fb58bfff700 1 -- 172.21.15.90:0/2420878741 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fb594027070 con 0x7fb5a41011e0 2024-02-21T23:41:42.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.373+0000 7fb5a9759700 1 -- 172.21.15.90:0/2420878741 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fb5a40623a0 con 0x7fb5a41011e0 2024-02-21T23:41:42.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.376+0000 7fb58bfff700 1 -- 172.21.15.90:0/2420878741 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fb594017440 con 0x7fb5a41011e0 2024-02-21T23:41:42.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.379+0000 7fb5a9759700 1 -- 172.21.15.90:0/2420878741 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fb58c0387a0 msgr2=0x7fb58c03ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:42.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.379+0000 7fb5a9759700 1 --2- 172.21.15.90:0/2420878741 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fb58c0387a0 0x7fb58c03ac60 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb598006fd0 tx=0x7fb598006e40 comp rx=0 tx=0).stop 2024-02-21T23:41:42.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.379+0000 7fb5a9759700 1 -- 172.21.15.90:0/2420878741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5a41011e0 msgr2=0x7fb5a4195ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:42.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.379+0000 7fb5a9759700 1 --2- 172.21.15.90:0/2420878741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5a41011e0 0x7fb5a4195ad0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb594005b40 tx=0x7fb5940057e0 comp rx=0 tx=0).stop 2024-02-21T23:41:42.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.379+0000 7fb5a9759700 1 -- 172.21.15.90:0/2420878741 shutdown_connections 2024-02-21T23:41:42.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.379+0000 7fb5a9759700 1 --2- 172.21.15.90:0/2420878741 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fb58c0387a0 0x7fb58c03ac60 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:42.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.379+0000 7fb5a9759700 1 --2- 172.21.15.90:0/2420878741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5a41011e0 0x7fb5a4195ad0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:42.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.379+0000 7fb5a9759700 1 -- 172.21.15.90:0/2420878741 >> 172.21.15.90:0/2420878741 conn(0x7fb5a40fab70 msgr2=0x7fb5a40fb660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:42.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.379+0000 7fb5a9759700 1 -- 172.21.15.90:0/2420878741 shutdown_connections 2024-02-21T23:41:42.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:42.379+0000 7fb5a9759700 1 -- 172.21.15.90:0/2420878741 wait complete. 2024-02-21T23:41:42.746 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-02-21T23:41:42.747 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-02-21T23:41:42.977 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.a/config 2024-02-21T23:41:43.082 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:42 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:41:43.083 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:42 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:41:43.083 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:42 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi090", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:41:43.083 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:42 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:41:43.083 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:42 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:41:43.083 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:42 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:41:43.083 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:42 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/2420878741' entity='client.admin' 2024-02-21T23:41:43.083 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:42 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:41:43.083 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:42 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:41:44.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.053+0000 7fc784e1d700 1 -- 172.21.15.90:0/791222957 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7801006d0 msgr2=0x7fc780100af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:44.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.053+0000 7fc784e1d700 1 --2- 172.21.15.90:0/791222957 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7801006d0 0x7fc780100af0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc770009910 tx=0x7fc770009c20 comp rx=0 tx=0).stop 2024-02-21T23:41:44.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.053+0000 7fc784e1d700 1 -- 172.21.15.90:0/791222957 shutdown_connections 2024-02-21T23:41:44.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.053+0000 7fc784e1d700 1 --2- 172.21.15.90:0/791222957 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7801006d0 0x7fc780100af0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:44.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.053+0000 7fc784e1d700 1 -- 172.21.15.90:0/791222957 >> 172.21.15.90:0/791222957 conn(0x7fc7800fbc70 msgr2=0x7fc7800fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:44.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.053+0000 7fc784e1d700 1 -- 172.21.15.90:0/791222957 shutdown_connections 2024-02-21T23:41:44.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.053+0000 7fc784e1d700 1 -- 172.21.15.90:0/791222957 wait complete. 2024-02-21T23:41:44.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.054+0000 7fc784e1d700 1 Processor -- start 2024-02-21T23:41:44.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.054+0000 7fc784e1d700 1 -- start start 2024-02-21T23:41:44.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.054+0000 7fc784e1d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7801006d0 0x7fc780195b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:44.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.054+0000 7fc784e1d700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc780196090 con 0x7fc7801006d0 2024-02-21T23:41:44.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.054+0000 7fc77e59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7801006d0 0x7fc780195b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:44.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.054+0000 7fc77e59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7801006d0 0x7fc780195b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34286/0 (socket says 172.21.15.90:34286) 2024-02-21T23:41:44.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.054+0000 7fc77e59c700 1 -- 172.21.15.90:0/1757629445 learned_addr learned my addr 172.21.15.90:0/1757629445 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:44.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.055+0000 7fc77e59c700 1 -- 172.21.15.90:0/1757629445 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7700095f0 con 0x7fc7801006d0 2024-02-21T23:41:44.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.055+0000 7fc77e59c700 1 --2- 172.21.15.90:0/1757629445 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7801006d0 0x7fc780195b50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc770004f70 tx=0x7fc770004fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:44.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.055+0000 7fc76f7fe700 1 -- 172.21.15.90:0/1757629445 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc77001d070 con 0x7fc7801006d0 2024-02-21T23:41:44.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.055+0000 7fc76f7fe700 1 -- 172.21.15.90:0/1757629445 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc77000be20 con 0x7fc7801006d0 2024-02-21T23:41:44.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.055+0000 7fc784e1d700 1 -- 172.21.15.90:0/1757629445 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc780196290 con 0x7fc7801006d0 2024-02-21T23:41:44.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.055+0000 7fc76f7fe700 1 -- 172.21.15.90:0/1757629445 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc770022df0 con 0x7fc7801006d0 2024-02-21T23:41:44.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.055+0000 7fc784e1d700 1 -- 172.21.15.90:0/1757629445 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc780196730 con 0x7fc7801006d0 2024-02-21T23:41:44.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.056+0000 7fc76f7fe700 1 -- 172.21.15.90:0/1757629445 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) v1 ==== 45593+0+0 (secure 0 0 0) 0x7fc77000b960 con 0x7fc7801006d0 2024-02-21T23:41:44.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.056+0000 7fc784e1d700 1 -- 172.21.15.90:0/1757629445 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc78018f3f0 con 0x7fc7801006d0 2024-02-21T23:41:44.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.056+0000 7fc76f7fe700 1 --2- 172.21.15.90:0/1757629445 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fc7680387f0 0x7fc76803acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:44.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.056+0000 7fc76f7fe700 1 -- 172.21.15.90:0/1757629445 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc77004c970 con 0x7fc7801006d0 2024-02-21T23:41:44.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.056+0000 7fc77dd9b700 1 --2- 172.21.15.90:0/1757629445 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fc7680387f0 0x7fc76803acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:44.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.057+0000 7fc77dd9b700 1 --2- 172.21.15.90:0/1757629445 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fc7680387f0 0x7fc76803acb0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc774006fd0 tx=0x7fc774006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:44.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.061+0000 7fc76f7fe700 1 -- 172.21.15.90:0/1757629445 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fc77002bba0 con 0x7fc7801006d0 2024-02-21T23:41:44.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.255+0000 7fc784e1d700 1 -- 172.21.15.90:0/1757629445 --> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fc7800611b0 con 0x7fc7680387f0 2024-02-21T23:41:44.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.258+0000 7fc76f7fe700 1 -- 172.21.15.90:0/1757629445 <== mgr.14150 v2:172.21.15.90:6800/3697394685 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc7800611b0 con 0x7fc7680387f0 2024-02-21T23:41:44.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.260+0000 7fc784e1d700 1 -- 172.21.15.90:0/1757629445 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fc7680387f0 msgr2=0x7fc76803acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:44.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.260+0000 7fc784e1d700 1 --2- 172.21.15.90:0/1757629445 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fc7680387f0 0x7fc76803acb0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc774006fd0 tx=0x7fc774006e40 comp rx=0 tx=0).stop 2024-02-21T23:41:44.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.260+0000 7fc784e1d700 1 -- 172.21.15.90:0/1757629445 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7801006d0 msgr2=0x7fc780195b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:44.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.260+0000 7fc784e1d700 1 --2- 172.21.15.90:0/1757629445 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7801006d0 0x7fc780195b50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc770004f70 tx=0x7fc770004fa0 comp rx=0 tx=0).stop 2024-02-21T23:41:44.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.260+0000 7fc784e1d700 1 -- 172.21.15.90:0/1757629445 shutdown_connections 2024-02-21T23:41:44.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.260+0000 7fc784e1d700 1 --2- 172.21.15.90:0/1757629445 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fc7680387f0 0x7fc76803acb0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:44.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.260+0000 7fc784e1d700 1 --2- 172.21.15.90:0/1757629445 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7801006d0 0x7fc780195b50 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:44.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.260+0000 7fc784e1d700 1 -- 172.21.15.90:0/1757629445 >> 172.21.15.90:0/1757629445 conn(0x7fc7800fbc70 msgr2=0x7fc7800fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:44.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.260+0000 7fc784e1d700 1 -- 172.21.15.90:0/1757629445 shutdown_connections 2024-02-21T23:41:44.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:44.260+0000 7fc784e1d700 1 -- 172.21.15.90:0/1757629445 wait complete. 2024-02-21T23:41:44.674 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi176 2024-02-21T23:41:44.674 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-02-21T23:41:44.674 DEBUG:teuthology.orchestra.run.smithi176:> dd of=/etc/ceph/ceph.conf 2024-02-21T23:41:44.695 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-02-21T23:41:44.695 DEBUG:teuthology.orchestra.run.smithi176:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-21T23:41:44.755 INFO:tasks.cephadm:Adding host smithi176 to orchestrator... 2024-02-21T23:41:44.755 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph orch host add smithi176 2024-02-21T23:41:44.982 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.a/config 2024-02-21T23:41:45.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:45 smithi090 ceph-mon[43385]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:41:45.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:45 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:41:45.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:45 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:41:45.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:45 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:41:45.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:45 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:41:45.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:45 smithi090 ceph-mon[43385]: Updating smithi090:/etc/ceph/ceph.conf 2024-02-21T23:41:45.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:45 smithi090 ceph-mon[43385]: Updating smithi090:/var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/config/ceph.conf 2024-02-21T23:41:45.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:45 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:41:45.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:45 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:41:45.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:45 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:41:46.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:46.024+0000 7fecdebda700 1 -- 172.21.15.90:0/523162640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecd8106650 msgr2=0x7fecd8106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:46.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:46.024+0000 7fecdebda700 1 --2- 172.21.15.90:0/523162640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecd8106650 0x7fecd8106a30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7feccc009910 tx=0x7feccc009c20 comp rx=0 tx=0).stop 2024-02-21T23:41:46.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:46.024+0000 7fecdebda700 1 -- 172.21.15.90:0/523162640 shutdown_connections 2024-02-21T23:41:46.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:46.024+0000 7fecdebda700 1 --2- 172.21.15.90:0/523162640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecd8106650 0x7fecd8106a30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:46.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:46.024+0000 7fecdebda700 1 -- 172.21.15.90:0/523162640 >> 172.21.15.90:0/523162640 conn(0x7fecd80fc180 msgr2=0x7fecd80fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:46.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:46.024+0000 7fecdebda700 1 -- 172.21.15.90:0/523162640 shutdown_connections 2024-02-21T23:41:46.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:46.024+0000 7fecdebda700 1 -- 172.21.15.90:0/523162640 wait complete. 2024-02-21T23:41:46.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:46.025+0000 7fecdebda700 1 Processor -- start 2024-02-21T23:41:46.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:46.025+0000 7fecdebda700 1 -- start start 2024-02-21T23:41:46.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:46.025+0000 7fecdebda700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecd8106650 0x7fecd8195f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:46.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:46.025+0000 7fecdebda700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd81964b0 con 0x7fecd8106650 2024-02-21T23:41:46.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:46.025+0000 7fecdc976700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecd8106650 0x7fecd8195f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:46.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:46.025+0000 7fecdc976700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecd8106650 0x7fecd8195f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34304/0 (socket says 172.21.15.90:34304) 2024-02-21T23:41:46.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:46.025+0000 7fecdc976700 1 -- 172.21.15.90:0/1243671504 learned_addr learned my addr 172.21.15.90:0/1243671504 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:46.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:46.026+0000 7fecdc976700 1 -- 172.21.15.90:0/1243671504 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feccc0095f0 con 0x7fecd8106650 2024-02-21T23:41:46.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:46.026+0000 7fecdc976700 1 --2- 172.21.15.90:0/1243671504 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecd8106650 0x7fecd8195f70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7feccc004f40 tx=0x7feccc005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:46.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:46.026+0000 7fecd5ffb700 1 -- 172.21.15.90:0/1243671504 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feccc01d070 con 0x7fecd8106650 2024-02-21T23:41:46.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:46.026+0000 7fecd5ffb700 1 -- 172.21.15.90:0/1243671504 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feccc00bbb0 con 0x7fecd8106650 2024-02-21T23:41:46.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:46.026+0000 7fecd5ffb700 1 -- 172.21.15.90:0/1243671504 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feccc003810 con 0x7fecd8106650 2024-02-21T23:41:46.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:46.026+0000 7fecdebda700 1 -- 172.21.15.90:0/1243671504 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecd8196710 con 0x7fecd8106650 2024-02-21T23:41:46.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:46.026+0000 7fecdebda700 1 -- 172.21.15.90:0/1243671504 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecd8196b90 con 0x7fecd8106650 2024-02-21T23:41:46.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:46.027+0000 7fecd5ffb700 1 -- 172.21.15.90:0/1243671504 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) v1 ==== 45593+0+0 (secure 0 0 0) 0x7feccc00bd20 con 0x7fecd8106650 2024-02-21T23:41:46.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:46.027+0000 7fecdebda700 1 -- 172.21.15.90:0/1243671504 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecd8101d90 con 0x7fecd8106650 2024-02-21T23:41:46.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:46.027+0000 7fecd5ffb700 1 --2- 172.21.15.90:0/1243671504 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fecc80387f0 0x7fecc803acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:46.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:46.027+0000 7fecd5ffb700 1 -- 172.21.15.90:0/1243671504 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7feccc04c420 con 0x7fecd8106650 2024-02-21T23:41:46.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:46.027+0000 7fecd7fff700 1 --2- 172.21.15.90:0/1243671504 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fecc80387f0 0x7fecc803acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:46.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:46.028+0000 7fecd7fff700 1 --2- 172.21.15.90:0/1243671504 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fecc80387f0 0x7fecc803acb0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fecc4006fd0 tx=0x7fecc4006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:46.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:46.032+0000 7fecd5ffb700 1 -- 172.21.15.90:0/1243671504 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7feccc02ab60 con 0x7fecd8106650 2024-02-21T23:41:46.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:46.225+0000 7fecdebda700 1 -- 172.21.15.90:0/1243671504 --> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi176", "target": ["mon-mgr", ""]}) v1 -- 0x7fecd8108a00 con 0x7fecc80387f0 2024-02-21T23:41:46.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:46 smithi090 ceph-mon[43385]: Updating smithi090:/etc/ceph/ceph.client.admin.keyring 2024-02-21T23:41:46.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:46 smithi090 ceph-mon[43385]: Updating smithi090:/var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/config/ceph.client.admin.keyring 2024-02-21T23:41:47.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:47 smithi090 ceph-mon[43385]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi176", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:41:48.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:48.399+0000 7fecd5ffb700 1 -- 172.21.15.90:0/1243671504 <== mgr.14150 v2:172.21.15.90:6800/3697394685 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fecd8108a00 con 0x7fecc80387f0 2024-02-21T23:41:48.383 INFO:teuthology.orchestra.run.smithi090.stdout:Added host 'smithi176' with addr '172.21.15.176' 2024-02-21T23:41:48.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:48.401+0000 7fecdebda700 1 -- 172.21.15.90:0/1243671504 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fecc80387f0 msgr2=0x7fecc803acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:48.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:48.401+0000 7fecdebda700 1 --2- 172.21.15.90:0/1243671504 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fecc80387f0 0x7fecc803acb0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fecc4006fd0 tx=0x7fecc4006e40 comp rx=0 tx=0).stop 2024-02-21T23:41:48.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:48.401+0000 7fecdebda700 1 -- 172.21.15.90:0/1243671504 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecd8106650 msgr2=0x7fecd8195f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:48.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:48.401+0000 7fecdebda700 1 --2- 172.21.15.90:0/1243671504 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecd8106650 0x7fecd8195f70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7feccc004f40 tx=0x7feccc005020 comp rx=0 tx=0).stop 2024-02-21T23:41:48.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:48.401+0000 7fecdebda700 1 -- 172.21.15.90:0/1243671504 shutdown_connections 2024-02-21T23:41:48.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:48.401+0000 7fecdebda700 1 --2- 172.21.15.90:0/1243671504 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fecc80387f0 0x7fecc803acb0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:48.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:48.401+0000 7fecdebda700 1 --2- 172.21.15.90:0/1243671504 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecd8106650 0x7fecd8195f70 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:48.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:48.401+0000 7fecdebda700 1 -- 172.21.15.90:0/1243671504 >> 172.21.15.90:0/1243671504 conn(0x7fecd80fc180 msgr2=0x7fecd81082f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:48.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:48.401+0000 7fecdebda700 1 -- 172.21.15.90:0/1243671504 shutdown_connections 2024-02-21T23:41:48.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:48.401+0000 7fecdebda700 1 -- 172.21.15.90:0/1243671504 wait complete. 2024-02-21T23:41:48.557 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:48 smithi090 ceph-mon[43385]: Deploying cephadm binary to smithi176 2024-02-21T23:41:48.762 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph orch host ls --format=json 2024-02-21T23:41:48.978 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.a/config 2024-02-21T23:41:49.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:49 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:41:49.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:49 smithi090 ceph-mon[43385]: Added host smithi176 2024-02-21T23:41:49.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:49 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:41:49.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:49 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:41:49.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:49 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:41:49.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:49.940+0000 7f83ca97e700 1 -- 172.21.15.90:0/1247965479 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83c4105620 msgr2=0x7f83c4105a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:49.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:49.940+0000 7f83ca97e700 1 --2- 172.21.15.90:0/1247965479 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83c4105620 0x7f83c4105a00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f83b4009910 tx=0x7f83b4009c20 comp rx=0 tx=0).stop 2024-02-21T23:41:49.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:49.940+0000 7f83ca97e700 1 -- 172.21.15.90:0/1247965479 shutdown_connections 2024-02-21T23:41:49.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:49.940+0000 7f83ca97e700 1 --2- 172.21.15.90:0/1247965479 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83c4105620 0x7f83c4105a00 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:49.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:49.940+0000 7f83ca97e700 1 -- 172.21.15.90:0/1247965479 >> 172.21.15.90:0/1247965479 conn(0x7f83c40750d0 msgr2=0x7f83c40754e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:49.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:49.941+0000 7f83ca97e700 1 -- 172.21.15.90:0/1247965479 shutdown_connections 2024-02-21T23:41:49.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:49.941+0000 7f83ca97e700 1 -- 172.21.15.90:0/1247965479 wait complete. 2024-02-21T23:41:49.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:49.941+0000 7f83ca97e700 1 Processor -- start 2024-02-21T23:41:49.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:49.941+0000 7f83ca97e700 1 -- start start 2024-02-21T23:41:49.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:49.942+0000 7f83ca97e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83c4105620 0x7f83c4195e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:49.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:49.942+0000 7f83ca97e700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c4196340 con 0x7f83c4105620 2024-02-21T23:41:49.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:49.942+0000 7f83c3fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83c4105620 0x7f83c4195e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:49.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:49.942+0000 7f83c3fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83c4105620 0x7f83c4195e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54290/0 (socket says 172.21.15.90:54290) 2024-02-21T23:41:49.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:49.942+0000 7f83c3fff700 1 -- 172.21.15.90:0/723408503 learned_addr learned my addr 172.21.15.90:0/723408503 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:49.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:49.942+0000 7f83c3fff700 1 -- 172.21.15.90:0/723408503 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b40095f0 con 0x7f83c4105620 2024-02-21T23:41:49.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:49.942+0000 7f83c3fff700 1 --2- 172.21.15.90:0/723408503 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83c4105620 0x7f83c4195e00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f83b4004f40 tx=0x7f83b4005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:49.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:49.942+0000 7f83c17fa700 1 -- 172.21.15.90:0/723408503 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f83b401d070 con 0x7f83c4105620 2024-02-21T23:41:49.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:49.942+0000 7f83c17fa700 1 -- 172.21.15.90:0/723408503 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f83b400bc10 con 0x7f83c4105620 2024-02-21T23:41:49.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:49.942+0000 7f83ca97e700 1 -- 172.21.15.90:0/723408503 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83c41965a0 con 0x7f83c4105620 2024-02-21T23:41:49.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:49.942+0000 7f83c17fa700 1 -- 172.21.15.90:0/723408503 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f83b40037d0 con 0x7f83c4105620 2024-02-21T23:41:49.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:49.942+0000 7f83ca97e700 1 -- 172.21.15.90:0/723408503 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83c4196aa0 con 0x7f83c4105620 2024-02-21T23:41:49.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:49.943+0000 7f83ca97e700 1 -- 172.21.15.90:0/723408503 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83c4100c90 con 0x7f83c4105620 2024-02-21T23:41:49.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:49.944+0000 7f83c17fa700 1 -- 172.21.15.90:0/723408503 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) v1 ==== 45593+0+0 (secure 0 0 0) 0x7f83b400b6f0 con 0x7f83c4105620 2024-02-21T23:41:49.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:49.944+0000 7f83c17fa700 1 --2- 172.21.15.90:0/723408503 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f83ac0387a0 0x7f83ac03ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:49.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:49.944+0000 7f83c17fa700 1 -- 172.21.15.90:0/723408503 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f83b404c730 con 0x7f83c4105620 2024-02-21T23:41:49.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:49.944+0000 7f83c37fe700 1 --2- 172.21.15.90:0/723408503 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f83ac0387a0 0x7f83ac03ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:49.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:49.944+0000 7f83c37fe700 1 --2- 172.21.15.90:0/723408503 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f83ac0387a0 0x7f83ac03ac60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f83b8006fd0 tx=0x7f83b8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:49.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:49.948+0000 7f83c17fa700 1 -- 172.21.15.90:0/723408503 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f83b4004440 con 0x7f83c4105620 2024-02-21T23:41:50.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:50.146+0000 7f83ca97e700 1 -- 172.21.15.90:0/723408503 --> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f83c4196aa0 con 0x7f83ac0387a0 2024-02-21T23:41:50.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:50.147+0000 7f83c17fa700 1 -- 172.21.15.90:0/723408503 <== mgr.14150 v2:172.21.15.90:6800/3697394685 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f83c4196aa0 con 0x7f83ac0387a0 2024-02-21T23:41:50.131 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:41:50.131 INFO:teuthology.orchestra.run.smithi090.stdout:[{"addr": "172.21.15.90", "hostname": "smithi090", "labels": [], "status": ""}, {"addr": "172.21.15.176", "hostname": "smithi176", "labels": [], "status": ""}] 2024-02-21T23:41:50.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:50.149+0000 7f83ca97e700 1 -- 172.21.15.90:0/723408503 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f83ac0387a0 msgr2=0x7f83ac03ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:50.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:50.149+0000 7f83ca97e700 1 --2- 172.21.15.90:0/723408503 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f83ac0387a0 0x7f83ac03ac60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f83b8006fd0 tx=0x7f83b8006e40 comp rx=0 tx=0).stop 2024-02-21T23:41:50.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:50.149+0000 7f83ca97e700 1 -- 172.21.15.90:0/723408503 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83c4105620 msgr2=0x7f83c4195e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:50.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:50.149+0000 7f83ca97e700 1 --2- 172.21.15.90:0/723408503 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83c4105620 0x7f83c4195e00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f83b4004f40 tx=0x7f83b4005020 comp rx=0 tx=0).stop 2024-02-21T23:41:50.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:50.149+0000 7f83ca97e700 1 -- 172.21.15.90:0/723408503 shutdown_connections 2024-02-21T23:41:50.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:50.149+0000 7f83ca97e700 1 --2- 172.21.15.90:0/723408503 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f83ac0387a0 0x7f83ac03ac60 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:50.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:50.149+0000 7f83ca97e700 1 --2- 172.21.15.90:0/723408503 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83c4105620 0x7f83c4195e00 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:50.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:50.149+0000 7f83ca97e700 1 -- 172.21.15.90:0/723408503 >> 172.21.15.90:0/723408503 conn(0x7f83c40750d0 msgr2=0x7f83c4103120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:50.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:50.149+0000 7f83ca97e700 1 -- 172.21.15.90:0/723408503 shutdown_connections 2024-02-21T23:41:50.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:50.149+0000 7f83ca97e700 1 -- 172.21.15.90:0/723408503 wait complete. 2024-02-21T23:41:50.512 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi178 2024-02-21T23:41:50.512 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-02-21T23:41:50.512 DEBUG:teuthology.orchestra.run.smithi178:> dd of=/etc/ceph/ceph.conf 2024-02-21T23:41:50.534 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-02-21T23:41:50.534 DEBUG:teuthology.orchestra.run.smithi178:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-21T23:41:50.594 INFO:tasks.cephadm:Adding host smithi178 to orchestrator... 2024-02-21T23:41:50.594 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph orch host add smithi178 2024-02-21T23:41:50.809 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.a/config 2024-02-21T23:41:50.955 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:50 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:41:50.955 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:50 smithi090 ceph-mon[43385]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-21T23:41:51.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:51.768+0000 7fd6ac539700 1 -- 172.21.15.90:0/1258106705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6a4106660 msgr2=0x7fd6a4106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:51.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:51.768+0000 7fd6ac539700 1 --2- 172.21.15.90:0/1258106705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6a4106660 0x7fd6a4106a40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0009910 tx=0x7fd6a0009c20 comp rx=0 tx=0).stop 2024-02-21T23:41:51.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:51.768+0000 7fd6ac539700 1 -- 172.21.15.90:0/1258106705 shutdown_connections 2024-02-21T23:41:51.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:51.768+0000 7fd6ac539700 1 --2- 172.21.15.90:0/1258106705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6a4106660 0x7fd6a4106a40 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:51.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:51.768+0000 7fd6ac539700 1 -- 172.21.15.90:0/1258106705 >> 172.21.15.90:0/1258106705 conn(0x7fd6a40fc170 msgr2=0x7fd6a40fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:51.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:51.768+0000 7fd6ac539700 1 -- 172.21.15.90:0/1258106705 shutdown_connections 2024-02-21T23:41:51.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:51.768+0000 7fd6ac539700 1 -- 172.21.15.90:0/1258106705 wait complete. 2024-02-21T23:41:51.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:51.769+0000 7fd6ac539700 1 Processor -- start 2024-02-21T23:41:51.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:51.769+0000 7fd6ac539700 1 -- start start 2024-02-21T23:41:51.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:51.769+0000 7fd6ac539700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6a4106660 0x7fd6a4195f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:51.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:51.769+0000 7fd6ac539700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a4196440 con 0x7fd6a4106660 2024-02-21T23:41:51.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:51.769+0000 7fd6aa2d5700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6a4106660 0x7fd6a4195f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:51.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:51.769+0000 7fd6aa2d5700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6a4106660 0x7fd6a4195f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54314/0 (socket says 172.21.15.90:54314) 2024-02-21T23:41:51.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:51.769+0000 7fd6aa2d5700 1 -- 172.21.15.90:0/2716361897 learned_addr learned my addr 172.21.15.90:0/2716361897 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:51.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:51.769+0000 7fd6aa2d5700 1 -- 172.21.15.90:0/2716361897 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a00095f0 con 0x7fd6a4106660 2024-02-21T23:41:51.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:51.769+0000 7fd6aa2d5700 1 --2- 172.21.15.90:0/2716361897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6a4106660 0x7fd6a4195f00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0004f40 tx=0x7fd6a0005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:51.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:51.770+0000 7fd69b7fe700 1 -- 172.21.15.90:0/2716361897 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd6a001d070 con 0x7fd6a4106660 2024-02-21T23:41:51.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:51.770+0000 7fd6ac539700 1 -- 172.21.15.90:0/2716361897 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6a41966a0 con 0x7fd6a4106660 2024-02-21T23:41:51.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:51.770+0000 7fd69b7fe700 1 -- 172.21.15.90:0/2716361897 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd6a000bbb0 con 0x7fd6a4106660 2024-02-21T23:41:51.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:51.770+0000 7fd69b7fe700 1 -- 172.21.15.90:0/2716361897 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd6a0003810 con 0x7fd6a4106660 2024-02-21T23:41:51.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:51.770+0000 7fd6ac539700 1 -- 172.21.15.90:0/2716361897 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6a4196b20 con 0x7fd6a4106660 2024-02-21T23:41:51.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:51.770+0000 7fd69b7fe700 1 -- 172.21.15.90:0/2716361897 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) v1 ==== 45593+0+0 (secure 0 0 0) 0x7fd6a0003a90 con 0x7fd6a4106660 2024-02-21T23:41:51.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:51.771+0000 7fd69b7fe700 1 --2- 172.21.15.90:0/2716361897 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fd6900387a0 0x7fd69003ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:51.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:51.771+0000 7fd6ac539700 1 -- 172.21.15.90:0/2716361897 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6a4104580 con 0x7fd6a4106660 2024-02-21T23:41:51.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:51.771+0000 7fd69b7fe700 1 -- 172.21.15.90:0/2716361897 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd6a004c620 con 0x7fd6a4106660 2024-02-21T23:41:51.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:51.771+0000 7fd6a9ad4700 1 --2- 172.21.15.90:0/2716361897 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fd6900387a0 0x7fd69003ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:51.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:51.771+0000 7fd6a9ad4700 1 --2- 172.21.15.90:0/2716361897 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fd6900387a0 0x7fd69003ac60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd694006fd0 tx=0x7fd694006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:51.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:51.776+0000 7fd69b7fe700 1 -- 172.21.15.90:0/2716361897 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fd6a0026020 con 0x7fd6a4106660 2024-02-21T23:41:51.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:51.979+0000 7fd6ac539700 1 -- 172.21.15.90:0/2716361897 --> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi178", "target": ["mon-mgr", ""]}) v1 -- 0x7fd6a41089d0 con 0x7fd6900387a0 2024-02-21T23:41:53.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:52 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:41:53.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:52 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:41:53.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:52 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:41:53.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:52 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:41:53.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:52 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi176", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:41:53.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:52 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:41:53.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:52 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:41:53.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:52 smithi090 ceph-mon[43385]: Updating smithi176:/etc/ceph/ceph.conf 2024-02-21T23:41:53.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:52 smithi090 ceph-mon[43385]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi178", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:41:53.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:52 smithi090 ceph-mon[43385]: Updating smithi176:/var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/config/ceph.conf 2024-02-21T23:41:53.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:52 smithi090 ceph-mon[43385]: Deploying cephadm binary to smithi178 2024-02-21T23:41:53.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:52 smithi090 ceph-mon[43385]: Updating smithi176:/etc/ceph/ceph.client.admin.keyring 2024-02-21T23:41:54.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:54.198+0000 7fd69b7fe700 1 -- 172.21.15.90:0/2716361897 <== mgr.14150 v2:172.21.15.90:6800/3697394685 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fd6a41089d0 con 0x7fd6900387a0 2024-02-21T23:41:54.183 INFO:teuthology.orchestra.run.smithi090.stdout:Added host 'smithi178' with addr '172.21.15.178' 2024-02-21T23:41:54.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:54.200+0000 7fd6ac539700 1 -- 172.21.15.90:0/2716361897 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fd6900387a0 msgr2=0x7fd69003ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:54.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:54.200+0000 7fd6ac539700 1 --2- 172.21.15.90:0/2716361897 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fd6900387a0 0x7fd69003ac60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd694006fd0 tx=0x7fd694006e40 comp rx=0 tx=0).stop 2024-02-21T23:41:54.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:54.201+0000 7fd6ac539700 1 -- 172.21.15.90:0/2716361897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6a4106660 msgr2=0x7fd6a4195f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:54.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:54.201+0000 7fd6ac539700 1 --2- 172.21.15.90:0/2716361897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6a4106660 0x7fd6a4195f00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0004f40 tx=0x7fd6a0005020 comp rx=0 tx=0).stop 2024-02-21T23:41:54.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:54.201+0000 7fd6ac539700 1 -- 172.21.15.90:0/2716361897 shutdown_connections 2024-02-21T23:41:54.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:54.201+0000 7fd6ac539700 1 --2- 172.21.15.90:0/2716361897 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fd6900387a0 0x7fd69003ac60 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:54.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:54.201+0000 7fd6ac539700 1 --2- 172.21.15.90:0/2716361897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6a4106660 0x7fd6a4195f00 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:54.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:54.201+0000 7fd6ac539700 1 -- 172.21.15.90:0/2716361897 >> 172.21.15.90:0/2716361897 conn(0x7fd6a40fc170 msgr2=0x7fd6a41082c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:54.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:54.201+0000 7fd6ac539700 1 -- 172.21.15.90:0/2716361897 shutdown_connections 2024-02-21T23:41:54.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:54.201+0000 7fd6ac539700 1 -- 172.21.15.90:0/2716361897 wait complete. 2024-02-21T23:41:54.348 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:54 smithi090 ceph-mon[43385]: Updating smithi176:/var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/config/ceph.client.admin.keyring 2024-02-21T23:41:54.348 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:54 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:41:54.348 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:54 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:41:54.348 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:54 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:41:54.349 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:54 smithi090 ceph-mon[43385]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:41:54.552 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph orch host ls --format=json 2024-02-21T23:41:54.768 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.a/config 2024-02-21T23:41:55.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:55 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:41:55.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:55 smithi090 ceph-mon[43385]: Added host smithi178 2024-02-21T23:41:55.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:55 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:41:55.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:55 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:41:55.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:55 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:41:55.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.758+0000 7fb05b59e700 1 -- 172.21.15.90:0/1700439455 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb05c105420 msgr2=0x7fb05c105800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:55.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.758+0000 7fb05b59e700 1 --2- 172.21.15.90:0/1700439455 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb05c105420 0x7fb05c105800 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb050009910 tx=0x7fb050009c20 comp rx=0 tx=0).stop 2024-02-21T23:41:55.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.758+0000 7fb05b59e700 1 -- 172.21.15.90:0/1700439455 shutdown_connections 2024-02-21T23:41:55.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.758+0000 7fb05b59e700 1 --2- 172.21.15.90:0/1700439455 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb05c105420 0x7fb05c105800 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:55.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.758+0000 7fb05b59e700 1 -- 172.21.15.90:0/1700439455 >> 172.21.15.90:0/1700439455 conn(0x7fb05c074ec0 msgr2=0x7fb05c0752d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:55.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.758+0000 7fb05b59e700 1 -- 172.21.15.90:0/1700439455 shutdown_connections 2024-02-21T23:41:55.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.758+0000 7fb05b59e700 1 -- 172.21.15.90:0/1700439455 wait complete. 2024-02-21T23:41:55.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.759+0000 7fb05b59e700 1 Processor -- start 2024-02-21T23:41:55.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.759+0000 7fb05b59e700 1 -- start start 2024-02-21T23:41:55.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.759+0000 7fb05b59e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb05c105420 0x7fb05c19b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:55.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.759+0000 7fb05b59e700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05c19be20 con 0x7fb05c105420 2024-02-21T23:41:55.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.759+0000 7fb05a59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb05c105420 0x7fb05c19b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:55.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.759+0000 7fb05a59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb05c105420 0x7fb05c19b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54342/0 (socket says 172.21.15.90:54342) 2024-02-21T23:41:55.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.759+0000 7fb05a59c700 1 -- 172.21.15.90:0/3974510575 learned_addr learned my addr 172.21.15.90:0/3974510575 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:55.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.759+0000 7fb05a59c700 1 -- 172.21.15.90:0/3974510575 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0500095f0 con 0x7fb05c105420 2024-02-21T23:41:55.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.759+0000 7fb05a59c700 1 --2- 172.21.15.90:0/3974510575 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb05c105420 0x7fb05c19b8e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb050004f40 tx=0x7fb050005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:55.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.760+0000 7fb04b7fe700 1 -- 172.21.15.90:0/3974510575 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb05001d070 con 0x7fb05c105420 2024-02-21T23:41:55.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.760+0000 7fb04b7fe700 1 -- 172.21.15.90:0/3974510575 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb050004370 con 0x7fb05c105420 2024-02-21T23:41:55.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.760+0000 7fb05b59e700 1 -- 172.21.15.90:0/3974510575 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb05c19c080 con 0x7fb05c105420 2024-02-21T23:41:55.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.760+0000 7fb04b7fe700 1 -- 172.21.15.90:0/3974510575 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb0500038e0 con 0x7fb05c105420 2024-02-21T23:41:55.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.760+0000 7fb05b59e700 1 -- 172.21.15.90:0/3974510575 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb05c195a70 con 0x7fb05c105420 2024-02-21T23:41:55.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.760+0000 7fb04b7fe700 1 -- 172.21.15.90:0/3974510575 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) v1 ==== 45593+0+0 (secure 0 0 0) 0x7fb050003ba0 con 0x7fb05c105420 2024-02-21T23:41:55.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.761+0000 7fb04b7fe700 1 --2- 172.21.15.90:0/3974510575 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fb0440387a0 0x7fb04403ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:55.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.761+0000 7fb05b59e700 1 -- 172.21.15.90:0/3974510575 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb05c100a60 con 0x7fb05c105420 2024-02-21T23:41:55.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.761+0000 7fb04b7fe700 1 -- 172.21.15.90:0/3974510575 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb05004c5f0 con 0x7fb05c105420 2024-02-21T23:41:55.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.761+0000 7fb059d9b700 1 --2- 172.21.15.90:0/3974510575 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fb0440387a0 0x7fb04403ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:55.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.761+0000 7fb059d9b700 1 --2- 172.21.15.90:0/3974510575 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fb0440387a0 0x7fb04403ac60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb04c006fd0 tx=0x7fb04c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:55.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.766+0000 7fb04b7fe700 1 -- 172.21.15.90:0/3974510575 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fb050026070 con 0x7fb05c105420 2024-02-21T23:41:55.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.959+0000 7fb05b59e700 1 -- 172.21.15.90:0/3974510575 --> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb05c196f50 con 0x7fb0440387a0 2024-02-21T23:41:55.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.960+0000 7fb04b7fe700 1 -- 172.21.15.90:0/3974510575 <== mgr.14150 v2:172.21.15.90:6800/3697394685 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+240 (secure 0 0 0) 0x7fb05c196f50 con 0x7fb0440387a0 2024-02-21T23:41:55.944 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:41:55.944 INFO:teuthology.orchestra.run.smithi090.stdout:[{"addr": "172.21.15.90", "hostname": "smithi090", "labels": [], "status": ""}, {"addr": "172.21.15.176", "hostname": "smithi176", "labels": [], "status": ""}, {"addr": "172.21.15.178", "hostname": "smithi178", "labels": [], "status": ""}] 2024-02-21T23:41:55.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.962+0000 7fb05b59e700 1 -- 172.21.15.90:0/3974510575 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fb0440387a0 msgr2=0x7fb04403ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:55.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.962+0000 7fb05b59e700 1 --2- 172.21.15.90:0/3974510575 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fb0440387a0 0x7fb04403ac60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb04c006fd0 tx=0x7fb04c006e40 comp rx=0 tx=0).stop 2024-02-21T23:41:55.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.962+0000 7fb05b59e700 1 -- 172.21.15.90:0/3974510575 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb05c105420 msgr2=0x7fb05c19b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:55.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.962+0000 7fb05b59e700 1 --2- 172.21.15.90:0/3974510575 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb05c105420 0x7fb05c19b8e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb050004f40 tx=0x7fb050005020 comp rx=0 tx=0).stop 2024-02-21T23:41:55.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.963+0000 7fb05b59e700 1 -- 172.21.15.90:0/3974510575 shutdown_connections 2024-02-21T23:41:55.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.963+0000 7fb05b59e700 1 --2- 172.21.15.90:0/3974510575 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fb0440387a0 0x7fb04403ac60 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:55.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.963+0000 7fb05b59e700 1 --2- 172.21.15.90:0/3974510575 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb05c105420 0x7fb05c19b8e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:55.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.963+0000 7fb05b59e700 1 -- 172.21.15.90:0/3974510575 >> 172.21.15.90:0/3974510575 conn(0x7fb05c074ec0 msgr2=0x7fb05c102f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:55.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.963+0000 7fb05b59e700 1 -- 172.21.15.90:0/3974510575 shutdown_connections 2024-02-21T23:41:55.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:55.963+0000 7fb05b59e700 1 -- 172.21.15.90:0/3974510575 wait complete. 2024-02-21T23:41:56.336 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-21T23:41:56.336 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph osd crush tunables default 2024-02-21T23:41:56.551 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.a/config 2024-02-21T23:41:56.821 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:56 smithi090 ceph-mon[43385]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:41:56.821 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:56 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:41:57.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:57.484+0000 7fa14f0a6700 1 -- 172.21.15.90:0/3199003251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa148106680 msgr2=0x7fa148106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:57.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:57.484+0000 7fa14f0a6700 1 --2- 172.21.15.90:0/3199003251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa148106680 0x7fa148106a60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa13c009910 tx=0x7fa13c009c20 comp rx=0 tx=0).stop 2024-02-21T23:41:57.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:57.484+0000 7fa14f0a6700 1 -- 172.21.15.90:0/3199003251 shutdown_connections 2024-02-21T23:41:57.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:57.484+0000 7fa14f0a6700 1 --2- 172.21.15.90:0/3199003251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa148106680 0x7fa148106a60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:57.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:57.484+0000 7fa14f0a6700 1 -- 172.21.15.90:0/3199003251 >> 172.21.15.90:0/3199003251 conn(0x7fa1480fc190 msgr2=0x7fa1480fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:57.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:57.484+0000 7fa14f0a6700 1 -- 172.21.15.90:0/3199003251 shutdown_connections 2024-02-21T23:41:57.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:57.484+0000 7fa14f0a6700 1 -- 172.21.15.90:0/3199003251 wait complete. 2024-02-21T23:41:57.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:57.484+0000 7fa14f0a6700 1 Processor -- start 2024-02-21T23:41:57.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:57.485+0000 7fa14f0a6700 1 -- start start 2024-02-21T23:41:57.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:57.485+0000 7fa14f0a6700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa148106680 0x7fa148195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:57.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:57.485+0000 7fa14f0a6700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa148196410 con 0x7fa148106680 2024-02-21T23:41:57.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:57.485+0000 7fa14ce42700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa148106680 0x7fa148195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:57.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:57.485+0000 7fa14ce42700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa148106680 0x7fa148195ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54360/0 (socket says 172.21.15.90:54360) 2024-02-21T23:41:57.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:57.485+0000 7fa14ce42700 1 -- 172.21.15.90:0/3952945997 learned_addr learned my addr 172.21.15.90:0/3952945997 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:41:57.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:57.485+0000 7fa14ce42700 1 -- 172.21.15.90:0/3952945997 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa13c0095f0 con 0x7fa148106680 2024-02-21T23:41:57.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:57.485+0000 7fa14ce42700 1 --2- 172.21.15.90:0/3952945997 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa148106680 0x7fa148195ed0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa13c004f40 tx=0x7fa13c005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:57.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:57.485+0000 7fa145ffb700 1 -- 172.21.15.90:0/3952945997 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa13c01d070 con 0x7fa148106680 2024-02-21T23:41:57.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:57.485+0000 7fa145ffb700 1 -- 172.21.15.90:0/3952945997 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa13c00bbb0 con 0x7fa148106680 2024-02-21T23:41:57.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:57.486+0000 7fa145ffb700 1 -- 172.21.15.90:0/3952945997 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa13c003810 con 0x7fa148106680 2024-02-21T23:41:57.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:57.486+0000 7fa14f0a6700 1 -- 172.21.15.90:0/3952945997 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa148196670 con 0x7fa148106680 2024-02-21T23:41:57.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:57.486+0000 7fa14f0a6700 1 -- 172.21.15.90:0/3952945997 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa148196af0 con 0x7fa148106680 2024-02-21T23:41:57.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:57.486+0000 7fa145ffb700 1 -- 172.21.15.90:0/3952945997 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) v1 ==== 45593+0+0 (secure 0 0 0) 0x7fa13c00bd20 con 0x7fa148106680 2024-02-21T23:41:57.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:57.487+0000 7fa14f0a6700 1 -- 172.21.15.90:0/3952945997 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa148101dc0 con 0x7fa148106680 2024-02-21T23:41:57.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:57.487+0000 7fa145ffb700 1 --2- 172.21.15.90:0/3952945997 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fa1380387f0 0x7fa13803acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:41:57.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:57.487+0000 7fa145ffb700 1 -- 172.21.15.90:0/3952945997 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa13c04c370 con 0x7fa148106680 2024-02-21T23:41:57.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:57.487+0000 7fa147fff700 1 --2- 172.21.15.90:0/3952945997 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fa1380387f0 0x7fa13803acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:41:57.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:57.487+0000 7fa147fff700 1 --2- 172.21.15.90:0/3952945997 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fa1380387f0 0x7fa13803acb0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa134006fd0 tx=0x7fa134006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:41:57.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:57.492+0000 7fa145ffb700 1 -- 172.21.15.90:0/3952945997 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fa13c02ab60 con 0x7fa148106680 2024-02-21T23:41:57.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:57.690+0000 7fa14f0a6700 1 -- 172.21.15.90:0/3952945997 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fa14819a7c0 con 0x7fa148106680 2024-02-21T23:41:57.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:57 smithi090 ceph-mon[43385]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-21T23:41:58.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:58.585+0000 7fa145ffb700 1 -- 172.21.15.90:0/3952945997 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fa13c026020 con 0x7fa148106680 2024-02-21T23:41:58.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:58.587+0000 7fa14f0a6700 1 -- 172.21.15.90:0/3952945997 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fa1380387f0 msgr2=0x7fa13803acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:58.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:58.587+0000 7fa14f0a6700 1 --2- 172.21.15.90:0/3952945997 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fa1380387f0 0x7fa13803acb0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa134006fd0 tx=0x7fa134006e40 comp rx=0 tx=0).stop 2024-02-21T23:41:58.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:58.587+0000 7fa14f0a6700 1 -- 172.21.15.90:0/3952945997 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa148106680 msgr2=0x7fa148195ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:41:58.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:58.587+0000 7fa14f0a6700 1 --2- 172.21.15.90:0/3952945997 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa148106680 0x7fa148195ed0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa13c004f40 tx=0x7fa13c005020 comp rx=0 tx=0).stop 2024-02-21T23:41:58.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:58.587+0000 7fa14f0a6700 1 -- 172.21.15.90:0/3952945997 shutdown_connections 2024-02-21T23:41:58.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:58.587+0000 7fa14f0a6700 1 --2- 172.21.15.90:0/3952945997 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fa1380387f0 0x7fa13803acb0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:58.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:58.587+0000 7fa14f0a6700 1 --2- 172.21.15.90:0/3952945997 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa148106680 0x7fa148195ed0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:41:58.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:58.587+0000 7fa14f0a6700 1 -- 172.21.15.90:0/3952945997 >> 172.21.15.90:0/3952945997 conn(0x7fa1480fc190 msgr2=0x7fa1481082c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:41:58.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:58.588+0000 7fa14f0a6700 1 -- 172.21.15.90:0/3952945997 shutdown_connections 2024-02-21T23:41:58.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:41:58.588+0000 7fa14f0a6700 1 -- 172.21.15.90:0/3952945997 wait complete. 2024-02-21T23:41:58.573 INFO:teuthology.orchestra.run.smithi090.stderr:adjusted tunables profile to default 2024-02-21T23:41:58.852 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:58 smithi090 ceph-mon[43385]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:41:58.852 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:58 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/3952945997' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-21T23:41:58.852 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:58 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:41:58.852 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:58 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:41:58.852 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:58 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:41:58.852 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:58 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:41:58.852 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:58 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi178", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:41:58.852 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:58 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:41:58.852 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:58 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:41:58.946 INFO:tasks.cephadm:Adding mon.a on smithi090 2024-02-21T23:41:58.946 INFO:tasks.cephadm:Adding mon.b on smithi176 2024-02-21T23:41:58.946 INFO:tasks.cephadm:Adding mon.c on smithi178 2024-02-21T23:41:58.946 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph orch apply mon '3;smithi090:172.21.15.90=a;smithi176:172.21.15.176=b;smithi178:172.21.15.178=c' 2024-02-21T23:41:59.180 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/config/ceph.conf 2024-02-21T23:41:59.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:59 smithi090 ceph-mon[43385]: Updating smithi178:/etc/ceph/ceph.conf 2024-02-21T23:41:59.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:59 smithi090 ceph-mon[43385]: Updating smithi178:/var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/config/ceph.conf 2024-02-21T23:41:59.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:59 smithi090 ceph-mon[43385]: Updating smithi178:/etc/ceph/ceph.client.admin.keyring 2024-02-21T23:41:59.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:59 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/3952945997' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-21T23:41:59.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:59 smithi090 ceph-mon[43385]: osdmap e4: 0 total, 0 up, 0 in 2024-02-21T23:41:59.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:59 smithi090 ceph-mon[43385]: Updating smithi178:/var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/config/ceph.client.admin.keyring 2024-02-21T23:41:59.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:59 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:41:59.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:59 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:41:59.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:41:59 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:00.228 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.244+0000 7f097e27d700 1 -- 172.21.15.178:0/708547877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0978105620 msgr2=0x7f0978105a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:42:00.228 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.244+0000 7f097e27d700 1 --2- 172.21.15.178:0/708547877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0978105620 0x7f0978105a00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0960009910 tx=0x7f0960009c20 comp rx=0 tx=0).stop 2024-02-21T23:42:00.229 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.244+0000 7f097e27d700 1 -- 172.21.15.178:0/708547877 shutdown_connections 2024-02-21T23:42:00.229 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.244+0000 7f097e27d700 1 --2- 172.21.15.178:0/708547877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0978105620 0x7f0978105a00 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:00.229 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.244+0000 7f097e27d700 1 -- 172.21.15.178:0/708547877 >> 172.21.15.178:0/708547877 conn(0x7f09780750d0 msgr2=0x7f09780754e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:42:00.229 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.244+0000 7f097e27d700 1 -- 172.21.15.178:0/708547877 shutdown_connections 2024-02-21T23:42:00.229 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.244+0000 7f097e27d700 1 -- 172.21.15.178:0/708547877 wait complete. 2024-02-21T23:42:00.230 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.245+0000 7f097e27d700 1 Processor -- start 2024-02-21T23:42:00.230 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.245+0000 7f097e27d700 1 -- start start 2024-02-21T23:42:00.230 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.245+0000 7f097e27d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0978105620 0x7f0978194d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:42:00.230 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.245+0000 7f097e27d700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0978195260 con 0x7f0978105620 2024-02-21T23:42:00.230 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.246+0000 7f0977fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0978105620 0x7f0978194d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:42:00.230 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.246+0000 7f0977fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0978105620 0x7f0978194d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.178:58204/0 (socket says 172.21.15.178:58204) 2024-02-21T23:42:00.230 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.246+0000 7f0977fff700 1 -- 172.21.15.178:0/719704564 learned_addr learned my addr 172.21.15.178:0/719704564 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-21T23:42:00.230 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.246+0000 7f0977fff700 1 -- 172.21.15.178:0/719704564 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09600095f0 con 0x7f0978105620 2024-02-21T23:42:00.232 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.246+0000 7f0977fff700 1 --2- 172.21.15.178:0/719704564 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0978105620 0x7f0978194d20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0960004f40 tx=0x7f0960005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:42:00.232 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.246+0000 7f09757fa700 1 -- 172.21.15.178:0/719704564 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f096001d070 con 0x7f0978105620 2024-02-21T23:42:00.232 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.246+0000 7f097e27d700 1 -- 172.21.15.178:0/719704564 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09781954c0 con 0x7f0978105620 2024-02-21T23:42:00.232 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.246+0000 7f09757fa700 1 -- 172.21.15.178:0/719704564 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f096000bc10 con 0x7f0978105620 2024-02-21T23:42:00.232 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.246+0000 7f09757fa700 1 -- 172.21.15.178:0/719704564 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f09600037d0 con 0x7f0978105620 2024-02-21T23:42:00.232 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.247+0000 7f097e27d700 1 -- 172.21.15.178:0/719704564 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09781959c0 con 0x7f0978105620 2024-02-21T23:42:00.232 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.247+0000 7f09757fa700 1 -- 172.21.15.178:0/719704564 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) v1 ==== 45593+0+0 (secure 0 0 0) 0x7f096000b6f0 con 0x7f0978105620 2024-02-21T23:42:00.232 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.247+0000 7f097e27d700 1 -- 172.21.15.178:0/719704564 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0978100c90 con 0x7f0978105620 2024-02-21T23:42:00.232 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.248+0000 7f09757fa700 1 --2- 172.21.15.178:0/719704564 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f09640387a0 0x7f096403ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:42:00.232 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.248+0000 7f09757fa700 1 -- 172.21.15.178:0/719704564 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f096004c1d0 con 0x7f0978105620 2024-02-21T23:42:00.232 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.248+0000 7f09777fe700 1 --2- 172.21.15.178:0/719704564 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f09640387a0 0x7f096403ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:42:00.233 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.248+0000 7f09777fe700 1 --2- 172.21.15.178:0/719704564 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f09640387a0 0x7f096403ac60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0968006fd0 tx=0x7f0968006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:42:00.237 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.253+0000 7f09757fa700 1 -- 172.21.15.178:0/719704564 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f0960004440 con 0x7f0978105620 2024-02-21T23:42:00.438 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.454+0000 7f097e27d700 1 -- 172.21.15.178:0/719704564 --> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi090:172.21.15.90=a;smithi176:172.21.15.176=b;smithi178:172.21.15.178=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f09780fd1a0 con 0x7f09640387a0 2024-02-21T23:42:00.445 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.461+0000 7f09757fa700 1 -- 172.21.15.178:0/719704564 <== mgr.14150 v2:172.21.15.90:6800/3697394685 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f09780fd1a0 con 0x7f09640387a0 2024-02-21T23:42:00.445 INFO:teuthology.orchestra.run.smithi178.stdout:Scheduled mon update... 2024-02-21T23:42:00.447 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.463+0000 7f097e27d700 1 -- 172.21.15.178:0/719704564 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f09640387a0 msgr2=0x7f096403ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:42:00.447 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.463+0000 7f097e27d700 1 --2- 172.21.15.178:0/719704564 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f09640387a0 0x7f096403ac60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0968006fd0 tx=0x7f0968006e40 comp rx=0 tx=0).stop 2024-02-21T23:42:00.447 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.463+0000 7f097e27d700 1 -- 172.21.15.178:0/719704564 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0978105620 msgr2=0x7f0978194d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:42:00.447 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.463+0000 7f097e27d700 1 --2- 172.21.15.178:0/719704564 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0978105620 0x7f0978194d20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0960004f40 tx=0x7f0960005020 comp rx=0 tx=0).stop 2024-02-21T23:42:00.447 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.463+0000 7f097e27d700 1 -- 172.21.15.178:0/719704564 shutdown_connections 2024-02-21T23:42:00.447 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.463+0000 7f097e27d700 1 --2- 172.21.15.178:0/719704564 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f09640387a0 0x7f096403ac60 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:00.447 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.463+0000 7f097e27d700 1 --2- 172.21.15.178:0/719704564 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0978105620 0x7f0978194d20 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:00.448 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.463+0000 7f097e27d700 1 -- 172.21.15.178:0/719704564 >> 172.21.15.178:0/719704564 conn(0x7f09780750d0 msgr2=0x7f09780fca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:42:00.448 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.463+0000 7f097e27d700 1 -- 172.21.15.178:0/719704564 shutdown_connections 2024-02-21T23:42:00.448 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:00.463+0000 7f097e27d700 1 -- 172.21.15.178:0/719704564 wait complete. 2024-02-21T23:42:00.864 DEBUG:teuthology.orchestra.run.smithi176:mon.b> sudo journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@mon.b.service 2024-02-21T23:42:00.868 DEBUG:teuthology.orchestra.run.smithi178:mon.c> sudo journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@mon.c.service 2024-02-21T23:42:00.871 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-21T23:42:00.871 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph mon dump -f json 2024-02-21T23:42:00.901 INFO:journalctl@ceph.mon.c.smithi178.stdout:-- Logs begin at Wed 2024-02-21 23:24:26 UTC. -- 2024-02-21T23:42:00.904 INFO:journalctl@ceph.mon.b.smithi176.stdout:-- Logs begin at Wed 2024-02-21 23:23:58 UTC. -- 2024-02-21T23:42:00.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:00 smithi090 ceph-mon[43385]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:00.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:00 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:00.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:00 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:42:00.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:00 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:00.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:00 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:42:00.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:00 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:00.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:00 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-21T23:42:00.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:00 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:01.604 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/config/ceph.conf 2024-02-21T23:42:01.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:01 smithi090 ceph-mon[43385]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi090:172.21.15.90=a;smithi176:172.21.15.176=b;smithi178:172.21.15.178=c", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:42:01.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:01 smithi090 ceph-mon[43385]: Saving service mon spec with placement smithi090:172.21.15.90=a;smithi176:172.21.15.176=b;smithi178:172.21.15.178=c;count:3 2024-02-21T23:42:01.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:01 smithi090 ceph-mon[43385]: Deploying daemon mon.c on smithi178 2024-02-21T23:42:02.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:02 smithi090 ceph-mon[43385]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:03.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.265+0000 7f995590a700 1 -- 172.21.15.178:0/211670864 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99501017b0 msgr2=0x7f9950101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:42:03.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.265+0000 7f995590a700 1 --2- 172.21.15.178:0/211670864 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99501017b0 0x7f9950101b90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9940009910 tx=0x7f9940009c20 comp rx=0 tx=0).stop 2024-02-21T23:42:03.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.265+0000 7f995590a700 1 -- 172.21.15.178:0/211670864 shutdown_connections 2024-02-21T23:42:03.251 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.265+0000 7f995590a700 1 --2- 172.21.15.178:0/211670864 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99501017b0 0x7f9950101b90 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:03.251 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.265+0000 7f995590a700 1 -- 172.21.15.178:0/211670864 >> 172.21.15.178:0/211670864 conn(0x7f99500fb0b0 msgr2=0x7f99500fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:42:03.251 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.265+0000 7f995590a700 1 -- 172.21.15.178:0/211670864 shutdown_connections 2024-02-21T23:42:03.251 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.265+0000 7f995590a700 1 -- 172.21.15.178:0/211670864 wait complete. 2024-02-21T23:42:03.251 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.266+0000 7f995590a700 1 Processor -- start 2024-02-21T23:42:03.251 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.266+0000 7f995590a700 1 -- start start 2024-02-21T23:42:03.252 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.267+0000 7f995590a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99501017b0 0x7f99500ff4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:42:03.252 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.267+0000 7f995590a700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9950101360 con 0x7f99501017b0 2024-02-21T23:42:03.252 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.267+0000 7f994effd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99501017b0 0x7f99500ff4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:42:03.252 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.267+0000 7f994effd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99501017b0 0x7f99500ff4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.178:58214/0 (socket says 172.21.15.178:58214) 2024-02-21T23:42:03.252 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.267+0000 7f994effd700 1 -- 172.21.15.178:0/1133770215 learned_addr learned my addr 172.21.15.178:0/1133770215 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-21T23:42:03.252 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.267+0000 7f994effd700 1 -- 172.21.15.178:0/1133770215 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99400095f0 con 0x7f99501017b0 2024-02-21T23:42:03.252 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.268+0000 7f994effd700 1 --2- 172.21.15.178:0/1133770215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99501017b0 0x7f99500ff4a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9940004f40 tx=0x7f9940005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:42:03.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.268+0000 7f9954908700 1 -- 172.21.15.178:0/1133770215 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f994001d070 con 0x7f99501017b0 2024-02-21T23:42:03.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.268+0000 7f9954908700 1 -- 172.21.15.178:0/1133770215 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f994000bc10 con 0x7f99501017b0 2024-02-21T23:42:03.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.268+0000 7f995590a700 1 -- 172.21.15.178:0/1133770215 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99500ff9e0 con 0x7f99501017b0 2024-02-21T23:42:03.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.268+0000 7f9954908700 1 -- 172.21.15.178:0/1133770215 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f99400037d0 con 0x7f99501017b0 2024-02-21T23:42:03.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.268+0000 7f995590a700 1 -- 172.21.15.178:0/1133770215 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99500ffe00 con 0x7f99501017b0 2024-02-21T23:42:03.254 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.269+0000 7f995590a700 1 -- 172.21.15.178:0/1133770215 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9950069370 con 0x7f99501017b0 2024-02-21T23:42:03.254 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.269+0000 7f9954908700 1 -- 172.21.15.178:0/1133770215 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) v1 ==== 45593+0+0 (secure 0 0 0) 0x7f9940003a40 con 0x7f99501017b0 2024-02-21T23:42:03.254 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.269+0000 7f9954908700 1 --2- 172.21.15.178:0/1133770215 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f99380387f0 0x7f993803acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:42:03.254 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.269+0000 7f9954908700 1 -- 172.21.15.178:0/1133770215 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f994004c3f0 con 0x7f99501017b0 2024-02-21T23:42:03.254 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.270+0000 7f994e7fc700 1 --2- 172.21.15.178:0/1133770215 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f99380387f0 0x7f993803acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:42:03.254 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.270+0000 7f994e7fc700 1 --2- 172.21.15.178:0/1133770215 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f99380387f0 0x7f993803acb0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9944006fd0 tx=0x7f9944006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:42:03.258 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.274+0000 7f9954908700 1 -- 172.21.15.178:0/1133770215 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f9940029360 con 0x7f99501017b0 2024-02-21T23:42:03.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.582+0000 7f995590a700 1 -- 172.21.15.178:0/1133770215 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9950100ac0 con 0x7f99501017b0 2024-02-21T23:42:03.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.583+0000 7f9954908700 1 -- 172.21.15.178:0/1133770215 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+736 (secure 0 0 0) 0x7f9950100ac0 con 0x7f99501017b0 2024-02-21T23:42:03.567 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-21T23:42:03.567 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"7ed81218-d112-11ee-95bd-87774f69a715","modified":"2024-02-21T23:40:39.758140Z","created":"2024-02-21T23:40:39.758140Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-21T23:42:03.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.585+0000 7f995590a700 1 -- 172.21.15.178:0/1133770215 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f99380387f0 msgr2=0x7f993803acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:42:03.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.585+0000 7f995590a700 1 --2- 172.21.15.178:0/1133770215 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f99380387f0 0x7f993803acb0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9944006fd0 tx=0x7f9944006e40 comp rx=0 tx=0).stop 2024-02-21T23:42:03.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.585+0000 7f995590a700 1 -- 172.21.15.178:0/1133770215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99501017b0 msgr2=0x7f99500ff4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:42:03.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.585+0000 7f995590a700 1 --2- 172.21.15.178:0/1133770215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99501017b0 0x7f99500ff4a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9940004f40 tx=0x7f9940005020 comp rx=0 tx=0).stop 2024-02-21T23:42:03.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.585+0000 7f995590a700 1 -- 172.21.15.178:0/1133770215 shutdown_connections 2024-02-21T23:42:03.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.585+0000 7f995590a700 1 --2- 172.21.15.178:0/1133770215 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f99380387f0 0x7f993803acb0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:03.570 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.585+0000 7f995590a700 1 --2- 172.21.15.178:0/1133770215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99501017b0 0x7f99500ff4a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:03.570 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.585+0000 7f995590a700 1 -- 172.21.15.178:0/1133770215 >> 172.21.15.178:0/1133770215 conn(0x7f99500fb0b0 msgr2=0x7f9950104600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:42:03.570 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.585+0000 7f995590a700 1 -- 172.21.15.178:0/1133770215 shutdown_connections 2024-02-21T23:42:03.570 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:03.585+0000 7f995590a700 1 -- 172.21.15.178:0/1133770215 wait complete. 2024-02-21T23:42:03.570 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-21T23:42:03.598 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 systemd[1]: Starting Ceph mon.c for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:42:03.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:03 smithi090 ceph-mon[43385]: from='client.? 172.21.15.178:0/1133770215' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-21T23:42:04.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: set uid:gid to 167:167 (ceph:ceph) 2024-02-21T23:42:04.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: ceph version 18.2.1-638-gc1dc18f4 (c1dc18f444603397768fcb7c34a8488b10284df6) reef (stable), process ceph-mon, pid 7 2024-02-21T23:42:04.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: pidfile_write: ignore empty --pid-file 2024-02-21T23:42:04.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: load: jerasure load: lrc 2024-02-21T23:42:04.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: RocksDB version: 7.9.2 2024-02-21T23:42:04.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Git sha 0 2024-02-21T23:42:04.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Compile date 2024-02-21 21:10:13 2024-02-21T23:42:04.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: DB SUMMARY 2024-02-21T23:42:04.426 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: DB Session ID: IG9H3RW0VX564MYQABNR 2024-02-21T23:42:04.426 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: CURRENT file: CURRENT 2024-02-21T23:42:04.426 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: IDENTITY file: IDENTITY 2024-02-21T23:42:04.426 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-02-21T23:42:04.426 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-02-21T23:42:04.426 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-02-21T23:42:04.426 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.error_if_exists: 0 2024-02-21T23:42:04.426 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.create_if_missing: 0 2024-02-21T23:42:04.426 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.paranoid_checks: 1 2024-02-21T23:42:04.426 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-21T23:42:04.426 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-21T23:42:04.426 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-21T23:42:04.426 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.env: 0x55a8a2355720 2024-02-21T23:42:04.427 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.fs: PosixFileSystem 2024-02-21T23:42:04.427 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.info_log: 0x55a8a54c34c0 2024-02-21T23:42:04.427 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.max_file_opening_threads: 16 2024-02-21T23:42:04.427 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.statistics: (nil) 2024-02-21T23:42:04.427 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.use_fsync: 0 2024-02-21T23:42:04.427 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.max_log_file_size: 0 2024-02-21T23:42:04.427 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-21T23:42:04.427 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-21T23:42:04.427 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.keep_log_file_num: 1000 2024-02-21T23:42:04.427 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.recycle_log_file_num: 0 2024-02-21T23:42:04.427 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.allow_fallocate: 1 2024-02-21T23:42:04.427 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.allow_mmap_reads: 0 2024-02-21T23:42:04.427 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.allow_mmap_writes: 0 2024-02-21T23:42:04.427 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.use_direct_reads: 0 2024-02-21T23:42:04.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-21T23:42:04.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.create_missing_column_families: 0 2024-02-21T23:42:04.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.db_log_dir: 2024-02-21T23:42:04.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.wal_dir: 2024-02-21T23:42:04.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-21T23:42:04.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-21T23:42:04.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-21T23:42:04.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-21T23:42:04.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-21T23:42:04.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-21T23:42:04.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.advise_random_on_open: 1 2024-02-21T23:42:04.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.db_write_buffer_size: 0 2024-02-21T23:42:04.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.write_buffer_manager: 0x55a8a4754320 2024-02-21T23:42:04.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-21T23:42:04.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-21T23:42:04.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-21T23:42:04.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.rate_limiter: (nil) 2024-02-21T23:42:04.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-21T23:42:04.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.wal_recovery_mode: 2 2024-02-21T23:42:04.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.enable_thread_tracking: 0 2024-02-21T23:42:04.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.enable_pipelined_write: 0 2024-02-21T23:42:04.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.unordered_write: 0 2024-02-21T23:42:04.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-21T23:42:04.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-21T23:42:04.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-21T23:42:04.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-21T23:42:04.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.row_cache: None 2024-02-21T23:42:04.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.wal_filter: None 2024-02-21T23:42:04.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-21T23:42:04.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.allow_ingest_behind: 0 2024-02-21T23:42:04.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.two_write_queues: 0 2024-02-21T23:42:04.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.manual_wal_flush: 0 2024-02-21T23:42:04.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.wal_compression: 0 2024-02-21T23:42:04.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.atomic_flush: 0 2024-02-21T23:42:04.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-21T23:42:04.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-21T23:42:04.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-21T23:42:04.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.log_readahead_size: 0 2024-02-21T23:42:04.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-21T23:42:04.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.best_efforts_recovery: 0 2024-02-21T23:42:04.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-21T23:42:04.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-21T23:42:04.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.allow_data_in_errors: 0 2024-02-21T23:42:04.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.db_host_id: __hostname__ 2024-02-21T23:42:04.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-21T23:42:04.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.max_background_jobs: 2 2024-02-21T23:42:04.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.max_background_compactions: -1 2024-02-21T23:42:04.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.max_subcompactions: 1 2024-02-21T23:42:04.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-21T23:42:04.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-21T23:42:04.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-21T23:42:04.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.max_total_wal_size: 0 2024-02-21T23:42:04.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-21T23:42:04.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-21T23:42:04.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-21T23:42:04.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-21T23:42:04.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.max_open_files: -1 2024-02-21T23:42:04.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.bytes_per_sync: 0 2024-02-21T23:42:04.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-21T23:42:04.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-21T23:42:04.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.compaction_readahead_size: 0 2024-02-21T23:42:04.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.max_background_flushes: -1 2024-02-21T23:42:04.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Compression algorithms supported: 2024-02-21T23:42:04.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-21T23:42:04.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: kZSTD supported: 0 2024-02-21T23:42:04.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: kXpressCompression supported: 0 2024-02-21T23:42:04.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: kLZ4HCCompression supported: 1 2024-02-21T23:42:04.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: kZlibCompression supported: 1 2024-02-21T23:42:04.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: kSnappyCompression supported: 1 2024-02-21T23:42:04.433 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: kLZ4Compression supported: 1 2024-02-21T23:42:04.433 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: kBZip2Compression supported: 0 2024-02-21T23:42:04.433 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-21T23:42:04.433 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-21T23:42:04.433 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-02-21T23:42:04.433 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-21T23:42:04.433 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-21T23:42:04.433 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.merge_operator: 2024-02-21T23:42:04.433 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.compaction_filter: None 2024-02-21T23:42:04.433 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.compaction_filter_factory: None 2024-02-21T23:42:04.433 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.sst_partitioner_factory: None 2024-02-21T23:42:04.433 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-21T23:42:04.433 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-21T23:42:04.433 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a8a54c35c0) 2024-02-21T23:42:04.434 INFO:journalctl@ceph.mon.c.smithi178.stdout: cache_index_and_filter_blocks: 1 2024-02-21T23:42:04.434 INFO:journalctl@ceph.mon.c.smithi178.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-21T23:42:04.434 INFO:journalctl@ceph.mon.c.smithi178.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-21T23:42:04.434 INFO:journalctl@ceph.mon.c.smithi178.stdout: pin_top_level_index_and_filter: 1 2024-02-21T23:42:04.434 INFO:journalctl@ceph.mon.c.smithi178.stdout: index_type: 0 2024-02-21T23:42:04.434 INFO:journalctl@ceph.mon.c.smithi178.stdout: data_block_index_type: 0 2024-02-21T23:42:04.434 INFO:journalctl@ceph.mon.c.smithi178.stdout: index_shortening: 1 2024-02-21T23:42:04.434 INFO:journalctl@ceph.mon.c.smithi178.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-21T23:42:04.434 INFO:journalctl@ceph.mon.c.smithi178.stdout: checksum: 4 2024-02-21T23:42:04.434 INFO:journalctl@ceph.mon.c.smithi178.stdout: no_block_cache: 0 2024-02-21T23:42:04.434 INFO:journalctl@ceph.mon.c.smithi178.stdout: block_cache: 0x55a8a47d51f0 2024-02-21T23:42:04.434 INFO:journalctl@ceph.mon.c.smithi178.stdout: block_cache_name: BinnedLRUCache 2024-02-21T23:42:04.434 INFO:journalctl@ceph.mon.c.smithi178.stdout: block_cache_options: 2024-02-21T23:42:04.434 INFO:journalctl@ceph.mon.c.smithi178.stdout: capacity : 536870912 2024-02-21T23:42:04.435 INFO:journalctl@ceph.mon.c.smithi178.stdout: num_shard_bits : 4 2024-02-21T23:42:04.435 INFO:journalctl@ceph.mon.c.smithi178.stdout: strict_capacity_limit : 0 2024-02-21T23:42:04.435 INFO:journalctl@ceph.mon.c.smithi178.stdout: high_pri_pool_ratio: 0.000 2024-02-21T23:42:04.435 INFO:journalctl@ceph.mon.c.smithi178.stdout: block_cache_compressed: (nil) 2024-02-21T23:42:04.435 INFO:journalctl@ceph.mon.c.smithi178.stdout: persistent_cache: (nil) 2024-02-21T23:42:04.435 INFO:journalctl@ceph.mon.c.smithi178.stdout: block_size: 4096 2024-02-21T23:42:04.435 INFO:journalctl@ceph.mon.c.smithi178.stdout: block_size_deviation: 10 2024-02-21T23:42:04.435 INFO:journalctl@ceph.mon.c.smithi178.stdout: block_restart_interval: 16 2024-02-21T23:42:04.435 INFO:journalctl@ceph.mon.c.smithi178.stdout: index_block_restart_interval: 1 2024-02-21T23:42:04.435 INFO:journalctl@ceph.mon.c.smithi178.stdout: metadata_block_size: 4096 2024-02-21T23:42:04.435 INFO:journalctl@ceph.mon.c.smithi178.stdout: partition_filters: 0 2024-02-21T23:42:04.435 INFO:journalctl@ceph.mon.c.smithi178.stdout: use_delta_encoding: 1 2024-02-21T23:42:04.435 INFO:journalctl@ceph.mon.c.smithi178.stdout: filter_policy: bloomfilter 2024-02-21T23:42:04.435 INFO:journalctl@ceph.mon.c.smithi178.stdout: whole_key_filtering: 1 2024-02-21T23:42:04.435 INFO:journalctl@ceph.mon.c.smithi178.stdout: verify_compression: 0 2024-02-21T23:42:04.436 INFO:journalctl@ceph.mon.c.smithi178.stdout: read_amp_bytes_per_bit: 0 2024-02-21T23:42:04.436 INFO:journalctl@ceph.mon.c.smithi178.stdout: format_version: 5 2024-02-21T23:42:04.436 INFO:journalctl@ceph.mon.c.smithi178.stdout: enable_index_compression: 1 2024-02-21T23:42:04.436 INFO:journalctl@ceph.mon.c.smithi178.stdout: block_align: 0 2024-02-21T23:42:04.436 INFO:journalctl@ceph.mon.c.smithi178.stdout: max_auto_readahead_size: 262144 2024-02-21T23:42:04.436 INFO:journalctl@ceph.mon.c.smithi178.stdout: prepopulate_block_cache: 0 2024-02-21T23:42:04.436 INFO:journalctl@ceph.mon.c.smithi178.stdout: initial_auto_readahead_size: 8192 2024-02-21T23:42:04.436 INFO:journalctl@ceph.mon.c.smithi178.stdout: num_file_reads_for_auto_readahead: 2 2024-02-21T23:42:04.436 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.write_buffer_size: 33554432 2024-02-21T23:42:04.436 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.max_write_buffer_number: 2 2024-02-21T23:42:04.436 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.compression: NoCompression 2024-02-21T23:42:04.436 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.bottommost_compression: Disabled 2024-02-21T23:42:04.436 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.prefix_extractor: nullptr 2024-02-21T23:42:04.436 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-21T23:42:04.436 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.num_levels: 7 2024-02-21T23:42:04.437 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-21T23:42:04.437 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-21T23:42:04.437 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-21T23:42:04.437 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-21T23:42:04.437 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-21T23:42:04.437 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-21T23:42:04.437 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-21T23:42:04.437 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-21T23:42:04.438 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-21T23:42:04.438 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-21T23:42:04.438 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-21T23:42:04.438 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-21T23:42:04.439 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-21T23:42:04.439 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.compression_opts.level: 32767 2024-02-21T23:42:04.439 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.compression_opts.strategy: 0 2024-02-21T23:42:04.439 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-21T23:42:04.439 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-21T23:42:04.439 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-21T23:42:04.439 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-21T23:42:04.439 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.compression_opts.enabled: false 2024-02-21T23:42:04.439 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-21T23:42:04.439 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-21T23:42:04.439 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-21T23:42:04.439 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-21T23:42:04.439 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.target_file_size_base: 67108864 2024-02-21T23:42:04.439 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-21T23:42:04.439 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-21T23:42:04.440 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-21T23:42:04.440 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-21T23:42:04.440 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-21T23:42:04.440 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-21T23:42:04.440 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-21T23:42:04.440 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-21T23:42:04.440 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-21T23:42:04.440 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-21T23:42:04.440 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-21T23:42:04.440 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-21T23:42:04.440 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-21T23:42:04.440 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-21T23:42:04.440 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.arena_block_size: 1048576 2024-02-21T23:42:04.440 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-21T23:42:04.441 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-21T23:42:04.441 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.disable_auto_compactions: 0 2024-02-21T23:42:04.441 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-21T23:42:04.441 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-21T23:42:04.441 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-21T23:42:04.441 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-21T23:42:04.441 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-21T23:42:04.441 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-21T23:42:04.441 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-21T23:42:04.441 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-21T23:42:04.441 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-21T23:42:04.441 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-21T23:42:04.441 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.table_properties_collectors: 2024-02-21T23:42:04.441 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.inplace_update_support: 0 2024-02-21T23:42:04.442 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-21T23:42:04.442 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:04 smithi178 bash[47487]: 63c25e6773c4390f2dceaa7858073e6e7304448eb17ce573e5dbd4ceab0587e9 2024-02-21T23:42:04.442 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-21T23:42:04.442 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-21T23:42:04.442 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-21T23:42:04.442 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.bloom_locality: 0 2024-02-21T23:42:04.442 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.max_successive_merges: 0 2024-02-21T23:42:04.442 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-21T23:42:04.442 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.paranoid_file_checks: 0 2024-02-21T23:42:04.442 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.force_consistency_checks: 1 2024-02-21T23:42:04.442 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.report_bg_io_stats: 0 2024-02-21T23:42:04.442 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.ttl: 2592000 2024-02-21T23:42:04.442 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-21T23:42:04.442 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-21T23:42:04.442 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-21T23:42:04.443 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.enable_blob_files: false 2024-02-21T23:42:04.443 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.min_blob_size: 0 2024-02-21T23:42:04.443 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.blob_file_size: 268435456 2024-02-21T23:42:04.443 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-21T23:42:04.443 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-21T23:42:04.443 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-21T23:42:04.443 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-21T23:42:04.443 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-21T23:42:04.443 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.blob_file_starting_level: 0 2024-02-21T23:42:04.443 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-21T23:42:04.443 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-21T23:42:04.443 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-02-21T23:42:04.443 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: e426deaa-f268-4ac9-8a11-993539620ecf 2024-02-21T23:42:04.443 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708558923995138, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-02-21T23:42:04.443 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-02-21T23:42:04.444 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708558923995566, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1708558923, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "e426deaa-f268-4ac9-8a11-993539620ecf", "db_session_id": "IG9H3RW0VX564MYQABNR", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-02-21T23:42:04.444 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708558923995638, "job": 1, "event": "recovery_finished"} 2024-02-21T23:42:04.444 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-02-21T23:42:04.444 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-21T23:42:04.444 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55a8a4872000 2024-02-21T23:42:04.444 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: DB pointer 0x55a8a4860000 2024-02-21T23:42:04.444 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-21T23:42:04.444 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-21T23:42:04.444 INFO:journalctl@ceph.mon.c.smithi178.stdout: ** DB Stats ** 2024-02-21T23:42:04.444 INFO:journalctl@ceph.mon.c.smithi178.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-21T23:42:04.445 INFO:journalctl@ceph.mon.c.smithi178.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-21T23:42:04.445 INFO:journalctl@ceph.mon.c.smithi178.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-21T23:42:04.445 INFO:journalctl@ceph.mon.c.smithi178.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-21T23:42:04.445 INFO:journalctl@ceph.mon.c.smithi178.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-21T23:42:04.445 INFO:journalctl@ceph.mon.c.smithi178.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-21T23:42:04.445 INFO:journalctl@ceph.mon.c.smithi178.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-21T23:42:04.445 INFO:journalctl@ceph.mon.c.smithi178.stdout: 2024-02-21T23:42:04.445 INFO:journalctl@ceph.mon.c.smithi178.stdout: ** Compaction Stats [default] ** 2024-02-21T23:42:04.445 INFO:journalctl@ceph.mon.c.smithi178.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-21T23:42:04.445 INFO:journalctl@ceph.mon.c.smithi178.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-21T23:42:04.445 INFO:journalctl@ceph.mon.c.smithi178.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-21T23:42:04.445 INFO:journalctl@ceph.mon.c.smithi178.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-21T23:42:04.445 INFO:journalctl@ceph.mon.c.smithi178.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-21T23:42:04.445 INFO:journalctl@ceph.mon.c.smithi178.stdout: 2024-02-21T23:42:04.446 INFO:journalctl@ceph.mon.c.smithi178.stdout: ** Compaction Stats [default] ** 2024-02-21T23:42:04.446 INFO:journalctl@ceph.mon.c.smithi178.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-21T23:42:04.446 INFO:journalctl@ceph.mon.c.smithi178.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-21T23:42:04.446 INFO:journalctl@ceph.mon.c.smithi178.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-21T23:42:04.446 INFO:journalctl@ceph.mon.c.smithi178.stdout: 2024-02-21T23:42:04.446 INFO:journalctl@ceph.mon.c.smithi178.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-21T23:42:04.446 INFO:journalctl@ceph.mon.c.smithi178.stdout: 2024-02-21T23:42:04.446 INFO:journalctl@ceph.mon.c.smithi178.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-21T23:42:04.446 INFO:journalctl@ceph.mon.c.smithi178.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-21T23:42:04.446 INFO:journalctl@ceph.mon.c.smithi178.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-21T23:42:04.446 INFO:journalctl@ceph.mon.c.smithi178.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-21T23:42:04.446 INFO:journalctl@ceph.mon.c.smithi178.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-21T23:42:04.446 INFO:journalctl@ceph.mon.c.smithi178.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-21T23:42:04.446 INFO:journalctl@ceph.mon.c.smithi178.stdout: Cumulative compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-21T23:42:04.446 INFO:journalctl@ceph.mon.c.smithi178.stdout: Interval compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-21T23:42:04.447 INFO:journalctl@ceph.mon.c.smithi178.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-21T23:42:04.447 INFO:journalctl@ceph.mon.c.smithi178.stdout: Block cache BinnedLRUCache@0x55a8a47d51f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-02-21T23:42:04.447 INFO:journalctl@ceph.mon.c.smithi178.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-02-21T23:42:04.447 INFO:journalctl@ceph.mon.c.smithi178.stdout: 2024-02-21T23:42:04.447 INFO:journalctl@ceph.mon.c.smithi178.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-21T23:42:04.447 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-02-21T23:42:04.447 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: using public_addr v2:172.21.15.178:0/0 -> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-02-21T23:42:04.447 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: starting mon.c rank -1 at public addrs [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] at bind addrs [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 7ed81218-d112-11ee-95bd-87774f69a715 2024-02-21T23:42:04.447 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:03 smithi178 ceph-mon[47547]: mon.c@-1(???) e0 preinit fsid 7ed81218-d112-11ee-95bd-87774f69a715 2024-02-21T23:42:04.447 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:04 smithi178 systemd[1]: Started Ceph mon.c for 7ed81218-d112-11ee-95bd-87774f69a715. 2024-02-21T23:42:04.447 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:04 smithi178 ceph-mon[47547]: mon.c@-1(synchronizing).mds e1 new map 2024-02-21T23:42:04.447 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:04 smithi178 ceph-mon[47547]: mon.c@-1(synchronizing).mds e1 print_map 2024-02-21T23:42:04.447 INFO:journalctl@ceph.mon.c.smithi178.stdout: e1 2024-02-21T23:42:04.447 INFO:journalctl@ceph.mon.c.smithi178.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-21T23:42:04.447 INFO:journalctl@ceph.mon.c.smithi178.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-21T23:42:04.448 INFO:journalctl@ceph.mon.c.smithi178.stdout: legacy client fscid: -1 2024-02-21T23:42:04.448 INFO:journalctl@ceph.mon.c.smithi178.stdout: 2024-02-21T23:42:04.448 INFO:journalctl@ceph.mon.c.smithi178.stdout: No filesystems configured 2024-02-21T23:42:04.448 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:04 smithi178 ceph-mon[47547]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-21T23:42:04.448 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:04 smithi178 ceph-mon[47547]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-21T23:42:04.448 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:04 smithi178 ceph-mon[47547]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-21T23:42:04.448 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:04 smithi178 ceph-mon[47547]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-21T23:42:04.448 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:04 smithi178 ceph-mon[47547]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-21T23:42:04.448 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:04 smithi178 ceph-mon[47547]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-21T23:42:04.448 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:04 smithi178 ceph-mon[47547]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-21T23:42:04.448 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:04 smithi178 ceph-mon[47547]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-21T23:42:04.448 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:04 smithi178 ceph-mon[47547]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-21T23:42:04.448 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:04 smithi178 ceph-mon[47547]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-21T23:42:04.449 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:04 smithi178 ceph-mon[47547]: Updating smithi178:/etc/ceph/ceph.conf 2024-02-21T23:42:04.449 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:04 smithi178 ceph-mon[47547]: Updating smithi178:/var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/config/ceph.conf 2024-02-21T23:42:04.449 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:04 smithi178 ceph-mon[47547]: Updating smithi178:/etc/ceph/ceph.client.admin.keyring 2024-02-21T23:42:04.449 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:04 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/3952945997' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-21T23:42:04.449 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:04 smithi178 ceph-mon[47547]: osdmap e4: 0 total, 0 up, 0 in 2024-02-21T23:42:04.449 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:04 smithi178 ceph-mon[47547]: Updating smithi178:/var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/config/ceph.client.admin.keyring 2024-02-21T23:42:04.449 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:04 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:04.449 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:04 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:04.449 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:04 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:04.449 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:04 smithi178 ceph-mon[47547]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:04.449 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:04 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:04.449 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:04 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:42:04.449 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:04 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:04.449 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:04 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:42:04.449 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:04 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:04.450 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:04 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-21T23:42:04.450 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:04 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:04.450 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:04 smithi178 ceph-mon[47547]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi090:172.21.15.90=a;smithi176:172.21.15.176=b;smithi178:172.21.15.178=c", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:42:04.450 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:04 smithi178 ceph-mon[47547]: Saving service mon spec with placement smithi090:172.21.15.90=a;smithi176:172.21.15.176=b;smithi178:172.21.15.178=c;count:3 2024-02-21T23:42:04.450 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:04 smithi178 ceph-mon[47547]: Deploying daemon mon.c on smithi178 2024-02-21T23:42:04.450 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:04 smithi178 ceph-mon[47547]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:04.450 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:04 smithi178 ceph-mon[47547]: from='client.? 172.21.15.178:0/1133770215' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-21T23:42:04.450 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:04 smithi178 ceph-mon[47547]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-02-21T23:42:05.037 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-21T23:42:05.038 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph mon dump -f json 2024-02-21T23:42:05.268 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.c/config 2024-02-21T23:42:09.369 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.385+0000 7fc26e923700 1 -- 172.21.15.178:0/306521282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2500035c0 msgr2=0x7fc250005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:42:09.369 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.385+0000 7fc26e923700 1 --2- 172.21.15.178:0/306521282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2500035c0 0x7fc250005a50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc2680ff5a0 tx=0x7fc25800feb0 comp rx=0 tx=0).stop 2024-02-21T23:42:09.369 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.385+0000 7fc26e923700 1 -- 172.21.15.178:0/306521282 shutdown_connections 2024-02-21T23:42:09.369 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.385+0000 7fc26e923700 1 --2- 172.21.15.178:0/306521282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2500035c0 0x7fc250005a50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:09.370 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.385+0000 7fc26e923700 1 -- 172.21.15.178:0/306521282 >> 172.21.15.178:0/306521282 conn(0x7fc2680763b0 msgr2=0x7fc2680fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:42:09.370 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.385+0000 7fc26e923700 1 -- 172.21.15.178:0/306521282 shutdown_connections 2024-02-21T23:42:09.370 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.385+0000 7fc26e923700 1 -- 172.21.15.178:0/306521282 wait complete. 2024-02-21T23:42:09.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.386+0000 7fc26e923700 1 Processor -- start 2024-02-21T23:42:09.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.386+0000 7fc26e923700 1 -- start start 2024-02-21T23:42:09.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.386+0000 7fc26e923700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc268194ca0 0x7fc268195080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:42:09.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.386+0000 7fc26e923700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2681955c0 0x7fc2681a2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:42:09.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.386+0000 7fc26e923700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc268195b00 con 0x7fc2681955c0 2024-02-21T23:42:09.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.386+0000 7fc26e923700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2680691c0 con 0x7fc268194ca0 2024-02-21T23:42:09.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.387+0000 7fc267fff700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc268194ca0 0x7fc268195080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:42:09.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.387+0000 7fc267fff700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc268194ca0 0x7fc268195080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:46698/0 (socket says 172.21.15.178:46698) 2024-02-21T23:42:09.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.387+0000 7fc267fff700 1 -- 172.21.15.178:0/3246163570 learned_addr learned my addr 172.21.15.178:0/3246163570 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-21T23:42:09.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.387+0000 7fc267fff700 1 -- 172.21.15.178:0/3246163570 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc268194ca0 msgr2=0x7fc268195080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-02-21T23:42:09.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.387+0000 7fc267fff700 1 -- 172.21.15.178:0/3246163570 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc268194ca0 msgr2=0x7fc268195080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-02-21T23:42:09.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.387+0000 7fc267fff700 1 --2- 172.21.15.178:0/3246163570 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc268194ca0 0x7fc268195080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-02-21T23:42:09.372 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.387+0000 7fc267fff700 1 --2- 172.21.15.178:0/3246163570 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc268194ca0 0x7fc268195080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-21T23:42:09.372 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.387+0000 7fc2677fe700 1 --2- 172.21.15.178:0/3246163570 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2681955c0 0x7fc2681a2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:42:09.373 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.387+0000 7fc2677fe700 1 -- 172.21.15.178:0/3246163570 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc268194ca0 msgr2=0x7fc268195080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-21T23:42:09.373 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.387+0000 7fc2677fe700 1 --2- 172.21.15.178:0/3246163570 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc268194ca0 0x7fc268195080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:09.373 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.387+0000 7fc2677fe700 1 -- 172.21.15.178:0/3246163570 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc25800f900 con 0x7fc2681955c0 2024-02-21T23:42:09.373 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.387+0000 7fc2677fe700 1 --2- 172.21.15.178:0/3246163570 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2681955c0 0x7fc2681a2310 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc25c00b4d0 tx=0x7fc25c00b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:42:09.373 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.388+0000 7fc2657fa700 1 -- 172.21.15.178:0/3246163570 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fc25c011630 con 0x7fc2681955c0 2024-02-21T23:42:09.373 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.388+0000 7fc2657fa700 1 -- 172.21.15.178:0/3246163570 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc25c011c50 con 0x7fc2681955c0 2024-02-21T23:42:09.374 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.388+0000 7fc26e923700 1 -- 172.21.15.178:0/3246163570 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2681a2850 con 0x7fc2681955c0 2024-02-21T23:42:09.374 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.388+0000 7fc2657fa700 1 -- 172.21.15.178:0/3246163570 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fc25c010d90 con 0x7fc2681955c0 2024-02-21T23:42:09.374 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.388+0000 7fc26e923700 1 -- 172.21.15.178:0/3246163570 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2681a2cc0 con 0x7fc2681955c0 2024-02-21T23:42:09.374 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.389+0000 7fc2657fa700 1 -- 172.21.15.178:0/3246163570 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) v1 ==== 45593+0+0 (secure 0 0 0) 0x7fc25c01a400 con 0x7fc2681955c0 2024-02-21T23:42:09.374 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.389+0000 7fc2657fa700 1 --2- 172.21.15.178:0/3246163570 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fc2540385d0 0x7fc25403aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:42:09.374 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.389+0000 7fc2657fa700 1 -- 172.21.15.178:0/3246163570 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc25c04ac50 con 0x7fc2681955c0 2024-02-21T23:42:09.374 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.389+0000 7fc26e923700 1 -- 172.21.15.178:0/3246163570 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc26818ed60 con 0x7fc2681955c0 2024-02-21T23:42:09.374 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.389+0000 7fc267fff700 1 --2- 172.21.15.178:0/3246163570 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fc2540385d0 0x7fc25403aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:42:09.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.390+0000 7fc267fff700 1 --2- 172.21.15.178:0/3246163570 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fc2540385d0 0x7fc25403aa90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc258011510 tx=0x7fc258005bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:42:09.379 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.395+0000 7fc2657fa700 1 -- 172.21.15.178:0/3246163570 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fc25c0178b0 con 0x7fc2681955c0 2024-02-21T23:42:09.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:09 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-21T23:42:09.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:09 smithi178 ceph-mon[47547]: mon.a calling monitor election 2024-02-21T23:42:09.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:09 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-21T23:42:09.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:09 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-21T23:42:09.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:09 smithi178 ceph-mon[47547]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:09.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:09 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-21T23:42:09.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:09 smithi178 ceph-mon[47547]: mon.c calling monitor election 2024-02-21T23:42:09.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:09 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-21T23:42:09.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:09 smithi178 ceph-mon[47547]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:09.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:09 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-21T23:42:09.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:09 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-21T23:42:09.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:09 smithi178 ceph-mon[47547]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-21T23:42:09.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:09 smithi178 ceph-mon[47547]: monmap e2: 2 mons at {a=[v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0],c=[v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-21T23:42:09.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:09 smithi178 ceph-mon[47547]: fsmap 2024-02-21T23:42:09.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:09 smithi178 ceph-mon[47547]: osdmap e4: 0 total, 0 up, 0 in 2024-02-21T23:42:09.426 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:09 smithi178 ceph-mon[47547]: mgrmap e13: x(active, since 35s) 2024-02-21T23:42:09.426 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:09 smithi178 ceph-mon[47547]: overall HEALTH_OK 2024-02-21T23:42:09.426 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:09 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:09.426 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:09 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:09.426 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:09 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:09.426 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:09 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-21T23:42:09.426 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:09 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:09.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:09 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-21T23:42:09.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:09 smithi090 ceph-mon[43385]: mon.a calling monitor election 2024-02-21T23:42:09.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:09 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-21T23:42:09.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:09 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-21T23:42:09.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:09 smithi090 ceph-mon[43385]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:09.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:09 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-21T23:42:09.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:09 smithi090 ceph-mon[43385]: mon.c calling monitor election 2024-02-21T23:42:09.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:09 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-21T23:42:09.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:09 smithi090 ceph-mon[43385]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:09.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:09 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-21T23:42:09.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:09 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-21T23:42:09.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:09 smithi090 ceph-mon[43385]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-21T23:42:09.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:09 smithi090 ceph-mon[43385]: monmap e2: 2 mons at {a=[v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0],c=[v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-21T23:42:09.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:09 smithi090 ceph-mon[43385]: fsmap 2024-02-21T23:42:09.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:09 smithi090 ceph-mon[43385]: osdmap e4: 0 total, 0 up, 0 in 2024-02-21T23:42:09.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:09 smithi090 ceph-mon[43385]: mgrmap e13: x(active, since 35s) 2024-02-21T23:42:09.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:09 smithi090 ceph-mon[43385]: overall HEALTH_OK 2024-02-21T23:42:09.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:09 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:09.480 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:09 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:09.480 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:09 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:09.480 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:09 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-21T23:42:09.480 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:09 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:09.650 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.666+0000 7fc26e923700 1 -- 172.21.15.178:0/3246163570 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc268068920 con 0x7fc2681955c0 2024-02-21T23:42:09.651 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.667+0000 7fc2657fa700 1 -- 172.21.15.178:0/3246163570 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1006 (secure 0 0 0) 0x7fc25c03d020 con 0x7fc2681955c0 2024-02-21T23:42:09.651 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-21T23:42:09.651 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":2,"fsid":"7ed81218-d112-11ee-95bd-87774f69a715","modified":"2024-02-21T23:42:04.017506Z","created":"2024-02-21T23:40:39.758140Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:3300","nonce":0},{"type":"v1","addr":"172.21.15.178:6789","nonce":0}]},"addr":"172.21.15.178:6789/0","public_addr":"172.21.15.178:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-21T23:42:09.653 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.669+0000 7fc26e923700 1 -- 172.21.15.178:0/3246163570 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fc2540385d0 msgr2=0x7fc25403aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:42:09.653 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.669+0000 7fc26e923700 1 --2- 172.21.15.178:0/3246163570 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fc2540385d0 0x7fc25403aa90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc258011510 tx=0x7fc258005bc0 comp rx=0 tx=0).stop 2024-02-21T23:42:09.653 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.669+0000 7fc26e923700 1 -- 172.21.15.178:0/3246163570 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2681955c0 msgr2=0x7fc2681a2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:42:09.653 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.669+0000 7fc26e923700 1 --2- 172.21.15.178:0/3246163570 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2681955c0 0x7fc2681a2310 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc25c00b4d0 tx=0x7fc25c00b890 comp rx=0 tx=0).stop 2024-02-21T23:42:09.653 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.669+0000 7fc26e923700 1 -- 172.21.15.178:0/3246163570 shutdown_connections 2024-02-21T23:42:09.653 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.669+0000 7fc26e923700 1 --2- 172.21.15.178:0/3246163570 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fc2540385d0 0x7fc25403aa90 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:09.654 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.669+0000 7fc26e923700 1 --2- 172.21.15.178:0/3246163570 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc268194ca0 0x7fc268195080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:09.654 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.669+0000 7fc26e923700 1 --2- 172.21.15.178:0/3246163570 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2681955c0 0x7fc2681a2310 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:09.654 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.669+0000 7fc26e923700 1 -- 172.21.15.178:0/3246163570 >> 172.21.15.178:0/3246163570 conn(0x7fc2500035c0 msgr2=0x7fc2680fc520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:42:09.654 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.669+0000 7fc26e923700 1 -- 172.21.15.178:0/3246163570 shutdown_connections 2024-02-21T23:42:09.654 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:09.669+0000 7fc26e923700 1 -- 172.21.15.178:0/3246163570 wait complete. 2024-02-21T23:42:09.654 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 2 2024-02-21T23:42:10.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:10 smithi178 ceph-mon[47547]: Deploying daemon mon.b on smithi176 2024-02-21T23:42:10.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:10 smithi178 ceph-mon[47547]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:10.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:10 smithi178 ceph-mon[47547]: from='client.? 172.21.15.178:0/3246163570' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-21T23:42:10.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:10 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-21T23:42:10.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:10 smithi090 ceph-mon[43385]: Deploying daemon mon.b on smithi176 2024-02-21T23:42:10.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:10 smithi090 ceph-mon[43385]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:10.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:10 smithi090 ceph-mon[43385]: from='client.? 172.21.15.178:0/3246163570' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-21T23:42:10.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:10 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-21T23:42:11.090 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-21T23:42:11.090 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph mon dump -f json 2024-02-21T23:42:11.317 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.c/config 2024-02-21T23:42:11.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 systemd[1]: Started Ceph mon.b for 7ed81218-d112-11ee-95bd-87774f69a715. 2024-02-21T23:42:11.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708558931536508, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1708558931, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "3ca20a19-c596-4b41-994e-ec1fd15351d1", "db_session_id": "NCLHFPMEEQPXXMLTSES3", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-02-21T23:42:11.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708558931536598, "job": 1, "event": "recovery_finished"} 2024-02-21T23:42:11.836 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-02-21T23:42:11.836 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-21T23:42:11.836 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55f26f1bc000 2024-02-21T23:42:11.836 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: rocksdb: DB pointer 0x55f26f1aa000 2024-02-21T23:42:11.836 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-21T23:42:11.837 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-21T23:42:11.837 INFO:journalctl@ceph.mon.b.smithi176.stdout: ** DB Stats ** 2024-02-21T23:42:11.837 INFO:journalctl@ceph.mon.b.smithi176.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-21T23:42:11.837 INFO:journalctl@ceph.mon.b.smithi176.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-21T23:42:11.837 INFO:journalctl@ceph.mon.b.smithi176.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-21T23:42:11.838 INFO:journalctl@ceph.mon.b.smithi176.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-21T23:42:11.838 INFO:journalctl@ceph.mon.b.smithi176.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-21T23:42:11.838 INFO:journalctl@ceph.mon.b.smithi176.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-21T23:42:11.838 INFO:journalctl@ceph.mon.b.smithi176.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-21T23:42:11.838 INFO:journalctl@ceph.mon.b.smithi176.stdout: 2024-02-21T23:42:11.838 INFO:journalctl@ceph.mon.b.smithi176.stdout: ** Compaction Stats [default] ** 2024-02-21T23:42:11.839 INFO:journalctl@ceph.mon.b.smithi176.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-21T23:42:11.839 INFO:journalctl@ceph.mon.b.smithi176.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-21T23:42:11.839 INFO:journalctl@ceph.mon.b.smithi176.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-21T23:42:11.839 INFO:journalctl@ceph.mon.b.smithi176.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-21T23:42:11.839 INFO:journalctl@ceph.mon.b.smithi176.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-21T23:42:11.840 INFO:journalctl@ceph.mon.b.smithi176.stdout: 2024-02-21T23:42:11.840 INFO:journalctl@ceph.mon.b.smithi176.stdout: ** Compaction Stats [default] ** 2024-02-21T23:42:11.840 INFO:journalctl@ceph.mon.b.smithi176.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-21T23:42:11.840 INFO:journalctl@ceph.mon.b.smithi176.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-21T23:42:11.840 INFO:journalctl@ceph.mon.b.smithi176.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-21T23:42:11.840 INFO:journalctl@ceph.mon.b.smithi176.stdout: 2024-02-21T23:42:11.841 INFO:journalctl@ceph.mon.b.smithi176.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-21T23:42:11.841 INFO:journalctl@ceph.mon.b.smithi176.stdout: 2024-02-21T23:42:11.841 INFO:journalctl@ceph.mon.b.smithi176.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-21T23:42:11.841 INFO:journalctl@ceph.mon.b.smithi176.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-21T23:42:11.841 INFO:journalctl@ceph.mon.b.smithi176.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-21T23:42:11.842 INFO:journalctl@ceph.mon.b.smithi176.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-21T23:42:11.842 INFO:journalctl@ceph.mon.b.smithi176.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-21T23:42:11.842 INFO:journalctl@ceph.mon.b.smithi176.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-21T23:42:11.842 INFO:journalctl@ceph.mon.b.smithi176.stdout: Cumulative compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-21T23:42:11.842 INFO:journalctl@ceph.mon.b.smithi176.stdout: Interval compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-21T23:42:11.842 INFO:journalctl@ceph.mon.b.smithi176.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-21T23:42:11.843 INFO:journalctl@ceph.mon.b.smithi176.stdout: Block cache BinnedLRUCache@0x55f26f11f1f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-02-21T23:42:11.843 INFO:journalctl@ceph.mon.b.smithi176.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-02-21T23:42:11.843 INFO:journalctl@ceph.mon.b.smithi176.stdout: 2024-02-21T23:42:11.843 INFO:journalctl@ceph.mon.b.smithi176.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-21T23:42:11.843 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-02-21T23:42:11.843 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: using public_addr v2:172.21.15.176:0/0 -> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-02-21T23:42:11.844 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: starting mon.b rank -1 at public addrs [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] at bind addrs [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 7ed81218-d112-11ee-95bd-87774f69a715 2024-02-21T23:42:11.844 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: mon.b@-1(???) e0 preinit fsid 7ed81218-d112-11ee-95bd-87774f69a715 2024-02-21T23:42:11.844 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: mon.b@-1(synchronizing).mds e1 new map 2024-02-21T23:42:11.844 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: mon.b@-1(synchronizing).mds e1 print_map 2024-02-21T23:42:11.844 INFO:journalctl@ceph.mon.b.smithi176.stdout: e1 2024-02-21T23:42:11.845 INFO:journalctl@ceph.mon.b.smithi176.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-21T23:42:11.845 INFO:journalctl@ceph.mon.b.smithi176.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-21T23:42:11.845 INFO:journalctl@ceph.mon.b.smithi176.stdout: legacy client fscid: -1 2024-02-21T23:42:11.845 INFO:journalctl@ceph.mon.b.smithi176.stdout: 2024-02-21T23:42:11.845 INFO:journalctl@ceph.mon.b.smithi176.stdout: No filesystems configured 2024-02-21T23:42:11.845 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-21T23:42:11.846 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-21T23:42:11.846 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-21T23:42:11.846 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-21T23:42:11.846 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-21T23:42:11.846 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-21T23:42:11.846 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-21T23:42:11.846 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-21T23:42:11.846 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-21T23:42:11.847 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-21T23:42:11.847 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: Updating smithi178:/etc/ceph/ceph.conf 2024-02-21T23:42:11.847 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: Updating smithi178:/var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/config/ceph.conf 2024-02-21T23:42:11.847 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: Updating smithi178:/etc/ceph/ceph.client.admin.keyring 2024-02-21T23:42:11.847 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/3952945997' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-21T23:42:11.847 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: osdmap e4: 0 total, 0 up, 0 in 2024-02-21T23:42:11.847 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: Updating smithi178:/var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/config/ceph.client.admin.keyring 2024-02-21T23:42:11.847 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:11.847 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:11.847 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:11.847 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:11.847 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:11.848 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:42:11.848 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:11.848 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:42:11.848 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:11.848 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-21T23:42:11.848 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:11.848 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi090:172.21.15.90=a;smithi176:172.21.15.176=b;smithi178:172.21.15.178=c", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:42:11.848 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: Saving service mon spec with placement smithi090:172.21.15.90=a;smithi176:172.21.15.176=b;smithi178:172.21.15.178=c;count:3 2024-02-21T23:42:11.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: Deploying daemon mon.c on smithi178 2024-02-21T23:42:11.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:11.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: from='client.? 172.21.15.178:0/1133770215' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-21T23:42:11.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-21T23:42:11.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: mon.a calling monitor election 2024-02-21T23:42:11.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-21T23:42:11.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-21T23:42:11.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:11.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-21T23:42:11.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: mon.c calling monitor election 2024-02-21T23:42:11.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-21T23:42:11.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:11.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-21T23:42:11.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-21T23:42:11.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-21T23:42:11.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: monmap e2: 2 mons at {a=[v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0],c=[v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-21T23:42:11.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: fsmap 2024-02-21T23:42:11.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: osdmap e4: 0 total, 0 up, 0 in 2024-02-21T23:42:11.851 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: mgrmap e13: x(active, since 35s) 2024-02-21T23:42:11.851 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: overall HEALTH_OK 2024-02-21T23:42:11.851 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:11.851 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:11.851 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:11.851 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-21T23:42:11.852 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:11.852 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: Deploying daemon mon.b on smithi176 2024-02-21T23:42:11.852 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:11.852 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: from='client.? 172.21.15.178:0/3246163570' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-21T23:42:11.852 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-21T23:42:11.852 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:11 smithi176 ceph-mon[47172]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-02-21T23:42:16.789 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.804+0000 7fdb0f662700 1 -- 172.21.15.178:0/2607897482 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb08106780 msgr2=0x7fdaf4005610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:42:16.789 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.804+0000 7fdb0f662700 1 --2- 172.21.15.178:0/2607897482 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb08106780 0x7fdaf4005610 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fdb04005fa0 tx=0x7fdb0400ff70 comp rx=0 tx=0).stop 2024-02-21T23:42:16.789 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.804+0000 7fdb0f662700 1 -- 172.21.15.178:0/2607897482 shutdown_connections 2024-02-21T23:42:16.790 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.804+0000 7fdb0f662700 1 --2- 172.21.15.178:0/2607897482 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb08106780 0x7fdaf4005610 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:16.790 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.804+0000 7fdb0f662700 1 -- 172.21.15.178:0/2607897482 >> 172.21.15.178:0/2607897482 conn(0x7fdb080fc270 msgr2=0x7fdb080fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:42:16.790 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.804+0000 7fdb0f662700 1 -- 172.21.15.178:0/2607897482 shutdown_connections 2024-02-21T23:42:16.790 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.804+0000 7fdb0f662700 1 -- 172.21.15.178:0/2607897482 wait complete. 2024-02-21T23:42:16.792 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.805+0000 7fdb0f662700 1 Processor -- start 2024-02-21T23:42:16.792 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.805+0000 7fdb0f662700 1 -- start start 2024-02-21T23:42:16.792 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.805+0000 7fdb0f662700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdb08106780 0x7fdb08196100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:42:16.792 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.805+0000 7fdb0f662700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb08196640 0x7fdb0819a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:42:16.792 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.805+0000 7fdb0f662700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdb0819af30 0x7fdb0819b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:42:16.792 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.805+0000 7fdb0f662700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb08196d30 con 0x7fdb08196640 2024-02-21T23:42:16.792 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.805+0000 7fdb0f662700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb081a2ca0 con 0x7fdb0819af30 2024-02-21T23:42:16.792 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.805+0000 7fdb0f662700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb081a2de0 con 0x7fdb08106780 2024-02-21T23:42:16.792 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.805+0000 7fdb0d3fe700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdb08106780 0x7fdb08196100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:42:16.792 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.805+0000 7fdb0d3fe700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdb08106780 0x7fdb08196100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:36144/0 (socket says 172.21.15.178:36144) 2024-02-21T23:42:16.793 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.805+0000 7fdb0d3fe700 1 -- 172.21.15.178:0/1250206240 learned_addr learned my addr 172.21.15.178:0/1250206240 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-21T23:42:16.793 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.805+0000 7fdb0cbfd700 1 --2- 172.21.15.178:0/1250206240 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb08196640 0x7fdb0819a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:42:16.793 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.805+0000 7fdb0d3fe700 1 -- 172.21.15.178:0/1250206240 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdb08106780 msgr2=0x7fdb08196100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-02-21T23:42:16.793 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.805+0000 7fdb0d3fe700 1 -- 172.21.15.178:0/1250206240 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdb08106780 msgr2=0x7fdb08196100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-02-21T23:42:16.793 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.805+0000 7fdb0d3fe700 1 --2- 172.21.15.178:0/1250206240 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdb08106780 0x7fdb08196100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-02-21T23:42:16.793 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.805+0000 7fdb0d3fe700 1 --2- 172.21.15.178:0/1250206240 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdb08106780 0x7fdb08196100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-21T23:42:16.793 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.805+0000 7fdb0dbff700 1 --2- 172.21.15.178:0/1250206240 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdb0819af30 0x7fdb0819b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:42:16.793 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.806+0000 7fdb0cbfd700 1 -- 172.21.15.178:0/1250206240 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdb08106780 msgr2=0x7fdb08196100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-21T23:42:16.793 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.806+0000 7fdb0cbfd700 1 --2- 172.21.15.178:0/1250206240 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdb08106780 0x7fdb08196100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:16.793 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.806+0000 7fdb0cbfd700 1 -- 172.21.15.178:0/1250206240 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdb0819af30 msgr2=0x7fdb0819b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:42:16.793 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.806+0000 7fdb0cbfd700 1 --2- 172.21.15.178:0/1250206240 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdb0819af30 0x7fdb0819b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:16.793 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.806+0000 7fdb0cbfd700 1 -- 172.21.15.178:0/1250206240 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb0400f970 con 0x7fdb08196640 2024-02-21T23:42:16.793 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.806+0000 7fdb0cbfd700 1 --2- 172.21.15.178:0/1250206240 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb08196640 0x7fdb0819a9f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fdaf800b550 tx=0x7fdaf800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:42:16.794 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.806+0000 7fdafe7fc700 1 -- 172.21.15.178:0/1250206240 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf8004230 con 0x7fdb08196640 2024-02-21T23:42:16.794 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.806+0000 7fdafe7fc700 1 -- 172.21.15.178:0/1250206240 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdaf8004390 con 0x7fdb08196640 2024-02-21T23:42:16.794 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.806+0000 7fdb0f662700 1 -- 172.21.15.178:0/1250206240 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb081a30c0 con 0x7fdb08196640 2024-02-21T23:42:16.794 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.806+0000 7fdafe7fc700 1 -- 172.21.15.178:0/1250206240 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf8011af0 con 0x7fdb08196640 2024-02-21T23:42:16.794 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.806+0000 7fdb0f662700 1 -- 172.21.15.178:0/1250206240 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb081a35c0 con 0x7fdb08196640 2024-02-21T23:42:16.794 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.807+0000 7fdafe7fc700 1 -- 172.21.15.178:0/1250206240 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) v1 ==== 45593+0+0 (secure 0 0 0) 0x7fdaf8011c50 con 0x7fdb08196640 2024-02-21T23:42:16.795 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.807+0000 7fdafe7fc700 1 --2- 172.21.15.178:0/1250206240 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fdaec038c70 0x7fdaec03b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:42:16.795 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.807+0000 7fdafe7fc700 1 -- 172.21.15.178:0/1250206240 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fdaf804ada0 con 0x7fdb08196640 2024-02-21T23:42:16.795 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.807+0000 7fdb0f662700 1 -- 172.21.15.178:0/1250206240 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb0818fd90 con 0x7fdb08196640 2024-02-21T23:42:16.795 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.808+0000 7fdb0d3fe700 1 --2- 172.21.15.178:0/1250206240 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fdaec038c70 0x7fdaec03b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:42:16.795 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.808+0000 7fdb0d3fe700 1 --2- 172.21.15.178:0/1250206240 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fdaec038c70 0x7fdaec03b130 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fdb04005850 tx=0x7fdb04005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:42:16.798 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:16.813+0000 7fdafe7fc700 1 -- 172.21.15.178:0/1250206240 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fdaf801a400 con 0x7fdb08196640 2024-02-21T23:42:16.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:16 smithi178 ceph-mon[47547]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:16.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:16 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-21T23:42:16.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:16 smithi178 ceph-mon[47547]: mon.a calling monitor election 2024-02-21T23:42:16.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:16 smithi178 ceph-mon[47547]: mon.c calling monitor election 2024-02-21T23:42:16.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:16 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-21T23:42:16.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:16 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-21T23:42:16.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:16 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-21T23:42:16.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:16 smithi178 ceph-mon[47547]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:16.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:16 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-21T23:42:16.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:16 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-21T23:42:16.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:16 smithi178 ceph-mon[47547]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:16.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:16 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-21T23:42:16.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:16 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-21T23:42:16.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:16 smithi178 ceph-mon[47547]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-21T23:42:16.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:16 smithi178 ceph-mon[47547]: monmap e3: 3 mons at {a=[v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0],b=[v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0],c=[v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-21T23:42:16.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:16 smithi178 ceph-mon[47547]: fsmap 2024-02-21T23:42:16.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:16 smithi178 ceph-mon[47547]: osdmap e4: 0 total, 0 up, 0 in 2024-02-21T23:42:16.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:16 smithi178 ceph-mon[47547]: mgrmap e13: x(active, since 43s) 2024-02-21T23:42:16.926 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:16 smithi178 ceph-mon[47547]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-02-21T23:42:16.926 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:16 smithi178 ceph-mon[47547]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-02-21T23:42:16.926 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:16 smithi178 ceph-mon[47547]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-02-21T23:42:16.926 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:16 smithi178 ceph-mon[47547]: mon.b (rank 2) addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] is down (out of quorum) 2024-02-21T23:42:16.926 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:16 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:16.926 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:16 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:16.926 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:16 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:16.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:16 smithi090 ceph-mon[43385]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:16.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:16 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-21T23:42:16.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:16 smithi090 ceph-mon[43385]: mon.a calling monitor election 2024-02-21T23:42:16.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:16 smithi090 ceph-mon[43385]: mon.c calling monitor election 2024-02-21T23:42:16.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:16 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-21T23:42:16.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:16 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-21T23:42:16.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:16 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-21T23:42:16.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:16 smithi090 ceph-mon[43385]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:16.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:16 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-21T23:42:16.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:16 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-21T23:42:16.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:16 smithi090 ceph-mon[43385]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:16.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:16 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-21T23:42:16.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:16 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-21T23:42:16.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:16 smithi090 ceph-mon[43385]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-21T23:42:16.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:16 smithi090 ceph-mon[43385]: monmap e3: 3 mons at {a=[v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0],b=[v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0],c=[v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-21T23:42:16.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:16 smithi090 ceph-mon[43385]: fsmap 2024-02-21T23:42:16.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:16 smithi090 ceph-mon[43385]: osdmap e4: 0 total, 0 up, 0 in 2024-02-21T23:42:16.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:16 smithi090 ceph-mon[43385]: mgrmap e13: x(active, since 43s) 2024-02-21T23:42:16.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:16 smithi090 ceph-mon[43385]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-02-21T23:42:16.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:16 smithi090 ceph-mon[43385]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-02-21T23:42:16.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:16 smithi090 ceph-mon[43385]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-02-21T23:42:16.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:16 smithi090 ceph-mon[43385]: mon.b (rank 2) addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] is down (out of quorum) 2024-02-21T23:42:16.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:16 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:16.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:16 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:16.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:16 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:17.095 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:17.109+0000 7fdb0f662700 1 -- 172.21.15.178:0/1250206240 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdb08101ec0 con 0x7fdb08196640 2024-02-21T23:42:17.095 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:17.110+0000 7fdafe7fc700 1 -- 172.21.15.178:0/1250206240 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1274 (secure 0 0 0) 0x7fdaf803e020 con 0x7fdb08196640 2024-02-21T23:42:17.096 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-21T23:42:17.096 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":3,"fsid":"7ed81218-d112-11ee-95bd-87774f69a715","modified":"2024-02-21T23:42:11.559100Z","created":"2024-02-21T23:40:39.758140Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:3300","nonce":0},{"type":"v1","addr":"172.21.15.178:6789","nonce":0}]},"addr":"172.21.15.178:6789/0","public_addr":"172.21.15.178:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-21T23:42:17.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:17.112+0000 7fdb0f662700 1 -- 172.21.15.178:0/1250206240 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fdaec038c70 msgr2=0x7fdaec03b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:42:17.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:17.112+0000 7fdb0f662700 1 --2- 172.21.15.178:0/1250206240 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fdaec038c70 0x7fdaec03b130 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fdb04005850 tx=0x7fdb04005da0 comp rx=0 tx=0).stop 2024-02-21T23:42:17.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:17.112+0000 7fdb0f662700 1 -- 172.21.15.178:0/1250206240 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb08196640 msgr2=0x7fdb0819a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:42:17.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:17.112+0000 7fdb0f662700 1 --2- 172.21.15.178:0/1250206240 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb08196640 0x7fdb0819a9f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fdaf800b550 tx=0x7fdaf800b910 comp rx=0 tx=0).stop 2024-02-21T23:42:17.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:17.112+0000 7fdb0f662700 1 -- 172.21.15.178:0/1250206240 shutdown_connections 2024-02-21T23:42:17.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:17.112+0000 7fdb0f662700 1 --2- 172.21.15.178:0/1250206240 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fdaec038c70 0x7fdaec03b130 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:17.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:17.112+0000 7fdb0f662700 1 --2- 172.21.15.178:0/1250206240 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdb0819af30 0x7fdb0819b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:17.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:17.112+0000 7fdb0f662700 1 --2- 172.21.15.178:0/1250206240 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdb08106780 0x7fdb08196100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:17.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:17.112+0000 7fdb0f662700 1 --2- 172.21.15.178:0/1250206240 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb08196640 0x7fdb0819a9f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:17.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:17.112+0000 7fdb0f662700 1 -- 172.21.15.178:0/1250206240 >> 172.21.15.178:0/1250206240 conn(0x7fdb080fc270 msgr2=0x7fdb080fd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:42:17.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:17.113+0000 7fdb0f662700 1 -- 172.21.15.178:0/1250206240 shutdown_connections 2024-02-21T23:42:17.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:17.113+0000 7fdb0f662700 1 -- 172.21.15.178:0/1250206240 wait complete. 2024-02-21T23:42:17.099 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 3 2024-02-21T23:42:17.863 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-02-21T23:42:17.863 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph config generate-minimal-conf 2024-02-21T23:42:17.888 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:17 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:17.888 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:17 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:42:17.889 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:17 smithi090 ceph-mon[43385]: from='client.? 172.21.15.178:0/1250206240' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-21T23:42:17.889 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:17 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:17.889 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:17 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:17.889 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:17 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-21T23:42:17.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:17 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:17.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:17 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:42:17.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:17 smithi178 ceph-mon[47547]: from='client.? 172.21.15.178:0/1250206240' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-21T23:42:17.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:17 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:17.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:17 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:17.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:17 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-21T23:42:18.088 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.a/config 2024-02-21T23:42:18.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:18 smithi176 ceph-mon[47172]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:18.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:18 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-21T23:42:18.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:18 smithi176 ceph-mon[47172]: mon.a calling monitor election 2024-02-21T23:42:18.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:18 smithi176 ceph-mon[47172]: mon.c calling monitor election 2024-02-21T23:42:18.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:18 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-21T23:42:18.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:18 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-21T23:42:18.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:18 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-21T23:42:18.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:18 smithi176 ceph-mon[47172]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:18.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:18 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-21T23:42:18.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:18 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-21T23:42:18.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:18 smithi176 ceph-mon[47172]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:18.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:18 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-21T23:42:18.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:18 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-21T23:42:18.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:18 smithi176 ceph-mon[47172]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-21T23:42:18.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:18 smithi176 ceph-mon[47172]: monmap e3: 3 mons at {a=[v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0],b=[v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0],c=[v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-21T23:42:18.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:18 smithi176 ceph-mon[47172]: fsmap 2024-02-21T23:42:18.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:18 smithi176 ceph-mon[47172]: osdmap e4: 0 total, 0 up, 0 in 2024-02-21T23:42:18.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:18 smithi176 ceph-mon[47172]: mgrmap e13: x(active, since 43s) 2024-02-21T23:42:18.836 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:18 smithi176 ceph-mon[47172]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-02-21T23:42:18.836 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:18 smithi176 ceph-mon[47172]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-02-21T23:42:18.836 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:18 smithi176 ceph-mon[47172]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-02-21T23:42:18.836 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:18 smithi176 ceph-mon[47172]: mon.b (rank 2) addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] is down (out of quorum) 2024-02-21T23:42:18.836 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:18 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:18.836 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:18 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:18.836 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:18 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:18.836 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:18 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:18.836 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:18 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:42:18.836 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:18 smithi176 ceph-mon[47172]: from='client.? 172.21.15.178:0/1250206240' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-21T23:42:18.836 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:18 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:18.836 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:18 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:18.836 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:18 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-21T23:42:19.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.131+0000 7fc3b8d90700 1 -- 172.21.15.90:0/2426615731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3b4106660 msgr2=0x7fc3b4106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:42:19.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.131+0000 7fc3b8d90700 1 --2- 172.21.15.90:0/2426615731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3b4106660 0x7fc3b4106a40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc39c009910 tx=0x7fc39c009c20 comp rx=0 tx=0).stop 2024-02-21T23:42:19.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.131+0000 7fc3b8d90700 1 -- 172.21.15.90:0/2426615731 shutdown_connections 2024-02-21T23:42:19.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.131+0000 7fc3b8d90700 1 --2- 172.21.15.90:0/2426615731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3b4106660 0x7fc3b4106a40 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:19.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.131+0000 7fc3b8d90700 1 -- 172.21.15.90:0/2426615731 >> 172.21.15.90:0/2426615731 conn(0x7fc3b40fc170 msgr2=0x7fc3b40fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:42:19.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.131+0000 7fc3b8d90700 1 -- 172.21.15.90:0/2426615731 shutdown_connections 2024-02-21T23:42:19.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.131+0000 7fc3b8d90700 1 -- 172.21.15.90:0/2426615731 wait complete. 2024-02-21T23:42:19.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.132+0000 7fc3b8d90700 1 Processor -- start 2024-02-21T23:42:19.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.132+0000 7fc3b8d90700 1 -- start start 2024-02-21T23:42:19.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.132+0000 7fc3b8d90700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc3b4106660 0x7fc3b4196070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:42:19.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.132+0000 7fc3b8d90700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3b41965b0 0x7fc3b419a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:42:19.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.132+0000 7fc3b8d90700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc3b419aea0 0x7fc3b419b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:42:19.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.132+0000 7fc3b8d90700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b4196ca0 con 0x7fc3b41965b0 2024-02-21T23:42:19.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.132+0000 7fc3b8d90700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b41a2c10 con 0x7fc3b4106660 2024-02-21T23:42:19.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.132+0000 7fc3b8d90700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b41a2d50 con 0x7fc3b419aea0 2024-02-21T23:42:19.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.132+0000 7fc3b1d9b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3b41965b0 0x7fc3b419a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:42:19.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.132+0000 7fc3b1d9b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3b41965b0 0x7fc3b419a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60824/0 (socket says 172.21.15.90:60824) 2024-02-21T23:42:19.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.132+0000 7fc3b1d9b700 1 -- 172.21.15.90:0/3671634101 learned_addr learned my addr 172.21.15.90:0/3671634101 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:42:19.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.132+0000 7fc3b259c700 1 --2- 172.21.15.90:0/3671634101 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc3b4106660 0x7fc3b4196070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:42:19.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.132+0000 7fc3b1d9b700 1 -- 172.21.15.90:0/3671634101 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc3b419aea0 msgr2=0x7fc3b419b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:42:19.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.132+0000 7fc3b1d9b700 1 --2- 172.21.15.90:0/3671634101 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc3b419aea0 0x7fc3b419b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:19.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.132+0000 7fc3b1d9b700 1 -- 172.21.15.90:0/3671634101 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc3b4106660 msgr2=0x7fc3b4196070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:42:19.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.132+0000 7fc3b1d9b700 1 --2- 172.21.15.90:0/3671634101 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc3b4106660 0x7fc3b4196070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:19.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.132+0000 7fc3b1d9b700 1 -- 172.21.15.90:0/3671634101 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc39c0095f0 con 0x7fc3b41965b0 2024-02-21T23:42:19.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.132+0000 7fc3b259c700 1 --2- 172.21.15.90:0/3671634101 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc3b4106660 0x7fc3b4196070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:42:19.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.133+0000 7fc3b1d9b700 1 --2- 172.21.15.90:0/3671634101 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3b41965b0 0x7fc3b419a960 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc3a400b780 tx=0x7fc3a400bb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:42:19.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.133+0000 7fc3ab7fe700 1 -- 172.21.15.90:0/3671634101 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a4004230 con 0x7fc3b41965b0 2024-02-21T23:42:19.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.133+0000 7fc3b8d90700 1 -- 172.21.15.90:0/3671634101 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3b41a2fd0 con 0x7fc3b41965b0 2024-02-21T23:42:19.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.133+0000 7fc3ab7fe700 1 -- 172.21.15.90:0/3671634101 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc3a4004390 con 0x7fc3b41965b0 2024-02-21T23:42:19.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.133+0000 7fc3ab7fe700 1 -- 172.21.15.90:0/3671634101 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a4011b30 con 0x7fc3b41965b0 2024-02-21T23:42:19.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.133+0000 7fc3b8d90700 1 -- 172.21.15.90:0/3671634101 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3b41a3520 con 0x7fc3b41965b0 2024-02-21T23:42:19.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.134+0000 7fc3b8d90700 1 -- 172.21.15.90:0/3671634101 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3b4101da0 con 0x7fc3b41965b0 2024-02-21T23:42:19.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.134+0000 7fc3ab7fe700 1 -- 172.21.15.90:0/3671634101 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) v1 ==== 45593+0+0 (secure 0 0 0) 0x7fc3a40036a0 con 0x7fc3b41965b0 2024-02-21T23:42:19.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.134+0000 7fc3ab7fe700 1 --2- 172.21.15.90:0/3671634101 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fc3a0038c20 0x7fc3a003b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:42:19.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.134+0000 7fc3ab7fe700 1 -- 172.21.15.90:0/3671634101 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc3a404b430 con 0x7fc3b41965b0 2024-02-21T23:42:19.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.134+0000 7fc3b259c700 1 --2- 172.21.15.90:0/3671634101 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fc3a0038c20 0x7fc3a003b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:42:19.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.135+0000 7fc3b259c700 1 --2- 172.21.15.90:0/3671634101 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fc3a0038c20 0x7fc3a003b0e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc39c00b5c0 tx=0x7fc39c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:42:19.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.139+0000 7fc3ab7fe700 1 -- 172.21.15.90:0/3671634101 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fc3a4004500 con 0x7fc3b41965b0 2024-02-21T23:42:19.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.339+0000 7fc3b8d90700 1 -- 172.21.15.90:0/3671634101 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fc3b40623a0 con 0x7fc3b41965b0 2024-02-21T23:42:19.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.340+0000 7fc3ab7fe700 1 -- 172.21.15.90:0/3671634101 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+275 (secure 0 0 0) 0x7fc3a4018b40 con 0x7fc3b41965b0 2024-02-21T23:42:19.325 INFO:teuthology.orchestra.run.smithi090.stdout:# minimal ceph.conf for 7ed81218-d112-11ee-95bd-87774f69a715 2024-02-21T23:42:19.325 INFO:teuthology.orchestra.run.smithi090.stdout:[global] 2024-02-21T23:42:19.325 INFO:teuthology.orchestra.run.smithi090.stdout: fsid = 7ed81218-d112-11ee-95bd-87774f69a715 2024-02-21T23:42:19.325 INFO:teuthology.orchestra.run.smithi090.stdout: mon_host = [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-02-21T23:42:19.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.342+0000 7fc3b8d90700 1 -- 172.21.15.90:0/3671634101 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fc3a0038c20 msgr2=0x7fc3a003b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:42:19.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.342+0000 7fc3b8d90700 1 --2- 172.21.15.90:0/3671634101 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fc3a0038c20 0x7fc3a003b0e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc39c00b5c0 tx=0x7fc39c005fd0 comp rx=0 tx=0).stop 2024-02-21T23:42:19.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.342+0000 7fc3b8d90700 1 -- 172.21.15.90:0/3671634101 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3b41965b0 msgr2=0x7fc3b419a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:42:19.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.342+0000 7fc3b8d90700 1 --2- 172.21.15.90:0/3671634101 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3b41965b0 0x7fc3b419a960 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc3a400b780 tx=0x7fc3a400bb40 comp rx=0 tx=0).stop 2024-02-21T23:42:19.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.342+0000 7fc3b8d90700 1 -- 172.21.15.90:0/3671634101 shutdown_connections 2024-02-21T23:42:19.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.342+0000 7fc3b8d90700 1 --2- 172.21.15.90:0/3671634101 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fc3a0038c20 0x7fc3a003b0e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:19.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.342+0000 7fc3b8d90700 1 --2- 172.21.15.90:0/3671634101 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc3b419aea0 0x7fc3b419b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:19.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.342+0000 7fc3b8d90700 1 --2- 172.21.15.90:0/3671634101 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc3b4106660 0x7fc3b4196070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:19.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.342+0000 7fc3b8d90700 1 --2- 172.21.15.90:0/3671634101 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3b41965b0 0x7fc3b419a960 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:19.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.342+0000 7fc3b8d90700 1 -- 172.21.15.90:0/3671634101 >> 172.21.15.90:0/3671634101 conn(0x7fc3b40fc170 msgr2=0x7fc3b4102160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:42:19.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.342+0000 7fc3b8d90700 1 -- 172.21.15.90:0/3671634101 shutdown_connections 2024-02-21T23:42:19.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:19.342+0000 7fc3b8d90700 1 -- 172.21.15.90:0/3671634101 wait complete. 2024-02-21T23:42:19.724 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:19 smithi176 ceph-mon[47172]: mon.b calling monitor election 2024-02-21T23:42:19.724 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:19 smithi176 ceph-mon[47172]: Updating smithi090:/etc/ceph/ceph.conf 2024-02-21T23:42:19.724 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:19 smithi176 ceph-mon[47172]: Updating smithi176:/etc/ceph/ceph.conf 2024-02-21T23:42:19.724 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:19 smithi176 ceph-mon[47172]: Updating smithi178:/etc/ceph/ceph.conf 2024-02-21T23:42:19.724 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:19 smithi176 ceph-mon[47172]: Updating smithi090:/var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/config/ceph.conf 2024-02-21T23:42:19.724 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:19 smithi176 ceph-mon[47172]: Updating smithi178:/var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/config/ceph.conf 2024-02-21T23:42:19.725 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:19 smithi176 ceph-mon[47172]: Updating smithi176:/var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/config/ceph.conf 2024-02-21T23:42:19.725 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:19 smithi176 ceph-mon[47172]: Reconfiguring mon.a (unknown last config time)... 2024-02-21T23:42:19.725 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:19 smithi176 ceph-mon[47172]: Reconfiguring daemon mon.a on smithi090 2024-02-21T23:42:19.725 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:19 smithi176 ceph-mon[47172]: mon.b calling monitor election 2024-02-21T23:42:19.725 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:19 smithi176 ceph-mon[47172]: mon.c calling monitor election 2024-02-21T23:42:19.725 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:19 smithi176 ceph-mon[47172]: mon.a calling monitor election 2024-02-21T23:42:19.725 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:19 smithi176 ceph-mon[47172]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-21T23:42:19.725 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:19 smithi176 ceph-mon[47172]: monmap e3: 3 mons at {a=[v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0],b=[v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0],c=[v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-21T23:42:19.725 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:19 smithi176 ceph-mon[47172]: fsmap 2024-02-21T23:42:19.725 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:19 smithi176 ceph-mon[47172]: osdmap e4: 0 total, 0 up, 0 in 2024-02-21T23:42:19.725 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:19 smithi176 ceph-mon[47172]: mgrmap e13: x(active, since 45s) 2024-02-21T23:42:19.725 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:19 smithi176 ceph-mon[47172]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-02-21T23:42:19.726 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:19 smithi176 ceph-mon[47172]: Cluster is now healthy 2024-02-21T23:42:19.726 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:19 smithi176 ceph-mon[47172]: overall HEALTH_OK 2024-02-21T23:42:19.726 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:19 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:19.726 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:19 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:19.726 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:19 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-21T23:42:19.726 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:19 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-21T23:42:19.726 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:19 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:19.726 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:19 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/3671634101' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:19.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:19 smithi090 ceph-mon[43385]: mon.b calling monitor election 2024-02-21T23:42:19.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:19 smithi090 ceph-mon[43385]: Updating smithi090:/etc/ceph/ceph.conf 2024-02-21T23:42:19.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:19 smithi090 ceph-mon[43385]: Updating smithi176:/etc/ceph/ceph.conf 2024-02-21T23:42:19.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:19 smithi090 ceph-mon[43385]: Updating smithi178:/etc/ceph/ceph.conf 2024-02-21T23:42:19.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:19 smithi090 ceph-mon[43385]: Updating smithi090:/var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/config/ceph.conf 2024-02-21T23:42:19.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:19 smithi090 ceph-mon[43385]: Updating smithi178:/var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/config/ceph.conf 2024-02-21T23:42:19.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:19 smithi090 ceph-mon[43385]: Updating smithi176:/var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/config/ceph.conf 2024-02-21T23:42:19.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:19 smithi090 ceph-mon[43385]: Reconfiguring mon.a (unknown last config time)... 2024-02-21T23:42:19.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:19 smithi090 ceph-mon[43385]: Reconfiguring daemon mon.a on smithi090 2024-02-21T23:42:19.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:19 smithi090 ceph-mon[43385]: mon.b calling monitor election 2024-02-21T23:42:19.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:19 smithi090 ceph-mon[43385]: mon.c calling monitor election 2024-02-21T23:42:19.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:19 smithi090 ceph-mon[43385]: mon.a calling monitor election 2024-02-21T23:42:19.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:19 smithi090 ceph-mon[43385]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-21T23:42:19.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:19 smithi090 ceph-mon[43385]: monmap e3: 3 mons at {a=[v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0],b=[v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0],c=[v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-21T23:42:19.729 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:19 smithi090 ceph-mon[43385]: fsmap 2024-02-21T23:42:19.729 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:19 smithi090 ceph-mon[43385]: osdmap e4: 0 total, 0 up, 0 in 2024-02-21T23:42:19.729 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:19 smithi090 ceph-mon[43385]: mgrmap e13: x(active, since 45s) 2024-02-21T23:42:19.729 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:19 smithi090 ceph-mon[43385]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-02-21T23:42:19.729 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:19 smithi090 ceph-mon[43385]: Cluster is now healthy 2024-02-21T23:42:19.729 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:19 smithi090 ceph-mon[43385]: overall HEALTH_OK 2024-02-21T23:42:19.729 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:19 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:19.729 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:19 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:19.729 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:19 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-21T23:42:19.729 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:19 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-21T23:42:19.729 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:19 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:19.729 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:19 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/3671634101' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:19.744 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-02-21T23:42:19.744 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-21T23:42:19.744 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/ceph.conf 2024-02-21T23:42:19.814 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-21T23:42:19.815 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-21T23:42:19.881 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-02-21T23:42:19.881 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/ceph/ceph.conf 2024-02-21T23:42:19.914 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-02-21T23:42:19.914 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-21T23:42:19.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:19 smithi178 ceph-mon[47547]: mon.b calling monitor election 2024-02-21T23:42:19.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:19 smithi178 ceph-mon[47547]: Updating smithi090:/etc/ceph/ceph.conf 2024-02-21T23:42:19.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:19 smithi178 ceph-mon[47547]: Updating smithi176:/etc/ceph/ceph.conf 2024-02-21T23:42:19.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:19 smithi178 ceph-mon[47547]: Updating smithi178:/etc/ceph/ceph.conf 2024-02-21T23:42:19.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:19 smithi178 ceph-mon[47547]: Updating smithi090:/var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/config/ceph.conf 2024-02-21T23:42:19.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:19 smithi178 ceph-mon[47547]: Updating smithi178:/var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/config/ceph.conf 2024-02-21T23:42:19.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:19 smithi178 ceph-mon[47547]: Updating smithi176:/var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/config/ceph.conf 2024-02-21T23:42:19.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:19 smithi178 ceph-mon[47547]: Reconfiguring mon.a (unknown last config time)... 2024-02-21T23:42:19.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:19 smithi178 ceph-mon[47547]: Reconfiguring daemon mon.a on smithi090 2024-02-21T23:42:19.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:19 smithi178 ceph-mon[47547]: mon.b calling monitor election 2024-02-21T23:42:19.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:19 smithi178 ceph-mon[47547]: mon.c calling monitor election 2024-02-21T23:42:19.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:19 smithi178 ceph-mon[47547]: mon.a calling monitor election 2024-02-21T23:42:19.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:19 smithi178 ceph-mon[47547]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-21T23:42:19.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:19 smithi178 ceph-mon[47547]: monmap e3: 3 mons at {a=[v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0],b=[v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0],c=[v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-21T23:42:19.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:19 smithi178 ceph-mon[47547]: fsmap 2024-02-21T23:42:19.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:19 smithi178 ceph-mon[47547]: osdmap e4: 0 total, 0 up, 0 in 2024-02-21T23:42:19.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:19 smithi178 ceph-mon[47547]: mgrmap e13: x(active, since 45s) 2024-02-21T23:42:19.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:19 smithi178 ceph-mon[47547]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-02-21T23:42:19.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:19 smithi178 ceph-mon[47547]: Cluster is now healthy 2024-02-21T23:42:19.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:19 smithi178 ceph-mon[47547]: overall HEALTH_OK 2024-02-21T23:42:19.926 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:19 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:19.926 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:19 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:19.926 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:19 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-21T23:42:19.926 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:19 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-21T23:42:19.926 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:19 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:19.926 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:19 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/3671634101' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:19.983 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-02-21T23:42:19.983 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/ceph/ceph.conf 2024-02-21T23:42:20.019 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-02-21T23:42:20.020 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-21T23:42:20.093 INFO:tasks.cephadm:Adding mgr.x on smithi090 2024-02-21T23:42:20.093 INFO:tasks.cephadm:Adding mgr.y on smithi176 2024-02-21T23:42:20.093 INFO:tasks.cephadm:Adding mgr.z on smithi178 2024-02-21T23:42:20.093 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph orch apply mgr '3;smithi090=x;smithi176=y;smithi178=z' 2024-02-21T23:42:20.355 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.c/config 2024-02-21T23:42:20.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:20 smithi090 ceph-mon[43385]: Reconfiguring mon.b (monmap changed)... 2024-02-21T23:42:20.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:20 smithi090 ceph-mon[43385]: Reconfiguring daemon mon.b on smithi176 2024-02-21T23:42:20.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:20 smithi090 ceph-mon[43385]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:20.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:20 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-21T23:42:20.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:20 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:20.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:20 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:20.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:20 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-21T23:42:20.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:20 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-21T23:42:20.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:20 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:20.734 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:20 smithi178 ceph-mon[47547]: Reconfiguring mon.b (monmap changed)... 2024-02-21T23:42:20.734 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:20 smithi178 ceph-mon[47547]: Reconfiguring daemon mon.b on smithi176 2024-02-21T23:42:20.734 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:20 smithi178 ceph-mon[47547]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:20.734 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:20 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-21T23:42:20.734 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:20 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:20.735 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:20 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:20.735 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:20 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-21T23:42:20.735 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:20 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-21T23:42:20.735 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:20 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:20.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:20 smithi176 ceph-mon[47172]: Reconfiguring mon.b (monmap changed)... 2024-02-21T23:42:20.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:20 smithi176 ceph-mon[47172]: Reconfiguring daemon mon.b on smithi176 2024-02-21T23:42:20.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:20 smithi176 ceph-mon[47172]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:20.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:20 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-21T23:42:20.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:20 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:20.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:20 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:20.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:20 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-21T23:42:20.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:20 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-21T23:42:20.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:20 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:21.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:21 smithi090 ceph-mon[43385]: Reconfiguring mon.c (monmap changed)... 2024-02-21T23:42:21.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:21 smithi090 ceph-mon[43385]: Reconfiguring daemon mon.c on smithi178 2024-02-21T23:42:21.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:21 smithi176 ceph-mon[47172]: Reconfiguring mon.c (monmap changed)... 2024-02-21T23:42:21.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:21 smithi176 ceph-mon[47172]: Reconfiguring daemon mon.c on smithi178 2024-02-21T23:42:21.861 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:21.875+0000 7f0594c7d700 1 -- 172.21.15.178:0/2881657113 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05901007e0 msgr2=0x7f0590100c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:42:21.861 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:21.875+0000 7f0594c7d700 1 --2- 172.21.15.178:0/2881657113 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05901007e0 0x7f0590100c00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0584009910 tx=0x7f0584009c20 comp rx=0 tx=0).stop 2024-02-21T23:42:21.861 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:21.875+0000 7f0594c7d700 1 -- 172.21.15.178:0/2881657113 shutdown_connections 2024-02-21T23:42:21.861 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:21.875+0000 7f0594c7d700 1 --2- 172.21.15.178:0/2881657113 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05901007e0 0x7f0590100c00 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:21.861 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:21.875+0000 7f0594c7d700 1 -- 172.21.15.178:0/2881657113 >> 172.21.15.178:0/2881657113 conn(0x7f05900fbda0 msgr2=0x7f05900fe1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:42:21.861 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:21.875+0000 7f0594c7d700 1 -- 172.21.15.178:0/2881657113 shutdown_connections 2024-02-21T23:42:21.861 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:21.875+0000 7f0594c7d700 1 -- 172.21.15.178:0/2881657113 wait complete. 2024-02-21T23:42:21.864 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:21.876+0000 7f0594c7d700 1 Processor -- start 2024-02-21T23:42:21.864 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:21.876+0000 7f0594c7d700 1 -- start start 2024-02-21T23:42:21.864 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:21.876+0000 7f0594c7d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f05901007e0 0x7f0590195d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:42:21.864 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:21.876+0000 7f0594c7d700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0590196240 0x7f059019b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:42:21.864 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:21.876+0000 7f0594c7d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05901966c0 0x7f059019d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:42:21.864 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:21.876+0000 7f0594c7d700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059019dfd0 con 0x7f05901966c0 2024-02-21T23:42:21.864 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:21.876+0000 7f0594c7d700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059019e140 con 0x7f05901007e0 2024-02-21T23:42:21.864 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:21.876+0000 7f0594c7d700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059019e300 con 0x7f0590196240 2024-02-21T23:42:21.864 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:21.876+0000 7f058dd9b700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0590196240 0x7f059019b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:42:21.865 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:21.877+0000 7f058dd9b700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0590196240 0x7f059019b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:36146/0 (socket says 172.21.15.178:36146) 2024-02-21T23:42:21.865 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:21.877+0000 7f058dd9b700 1 -- 172.21.15.178:0/122746529 learned_addr learned my addr 172.21.15.178:0/122746529 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-21T23:42:21.865 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:21.877+0000 7f058e59c700 1 --2- 172.21.15.178:0/122746529 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f05901007e0 0x7f0590195d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:42:21.865 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:21.877+0000 7f058ed9d700 1 --2- 172.21.15.178:0/122746529 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05901966c0 0x7f059019d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:42:21.865 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:21.877+0000 7f058dd9b700 1 -- 172.21.15.178:0/122746529 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f05901007e0 msgr2=0x7f0590195d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:42:21.865 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:21.877+0000 7f058dd9b700 1 --2- 172.21.15.178:0/122746529 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f05901007e0 0x7f0590195d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:21.866 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:21.877+0000 7f058dd9b700 1 -- 172.21.15.178:0/122746529 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05901966c0 msgr2=0x7f059019d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:42:21.866 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:21.877+0000 7f058dd9b700 1 --2- 172.21.15.178:0/122746529 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05901966c0 0x7f059019d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:21.866 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:21.877+0000 7f058dd9b700 1 -- 172.21.15.178:0/122746529 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05840095f0 con 0x7f0590196240 2024-02-21T23:42:21.866 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:21.877+0000 7f058dd9b700 1 --2- 172.21.15.178:0/122746529 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0590196240 0x7f059019b250 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f058000b780 tx=0x7f058000ba90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:42:21.866 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:21.877+0000 7f057f7fe700 1 -- 172.21.15.178:0/122746529 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0580004230 con 0x7f0590196240 2024-02-21T23:42:21.866 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:21.877+0000 7f0594c7d700 1 -- 172.21.15.178:0/122746529 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f059019e5e0 con 0x7f0590196240 2024-02-21T23:42:21.866 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:21.877+0000 7f057f7fe700 1 -- 172.21.15.178:0/122746529 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0580004390 con 0x7f0590196240 2024-02-21T23:42:21.866 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:21.877+0000 7f057f7fe700 1 -- 172.21.15.178:0/122746529 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0580011ad0 con 0x7f0590196240 2024-02-21T23:42:21.866 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:21.877+0000 7f0594c7d700 1 -- 172.21.15.178:0/122746529 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f059019eb30 con 0x7f0590196240 2024-02-21T23:42:21.866 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:21.878+0000 7f0594c7d700 1 -- 172.21.15.178:0/122746529 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f059018f4b0 con 0x7f0590196240 2024-02-21T23:42:21.866 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:21.878+0000 7f057f7fe700 1 -- 172.21.15.178:0/122746529 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 13) v1 ==== 45593+0+0 (secure 0 0 0) 0x7f05800036a0 con 0x7f0590196240 2024-02-21T23:42:21.866 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:21.879+0000 7f057f7fe700 1 --2- 172.21.15.178:0/122746529 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f0578038c20 0x7f057803b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:42:21.866 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:21.879+0000 7f057f7fe700 1 -- 172.21.15.178:0/122746529 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f058004b430 con 0x7f0590196240 2024-02-21T23:42:21.867 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:21.879+0000 7f058e59c700 1 --2- 172.21.15.178:0/122746529 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f0578038c20 0x7f057803b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:42:21.867 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:21.879+0000 7f058e59c700 1 --2- 172.21.15.178:0/122746529 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f0578038c20 0x7f057803b0e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f058400b5c0 tx=0x7f0584005200 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:42:21.868 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:21.883+0000 7f057f7fe700 1 -- 172.21.15.178:0/122746529 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f0580004500 con 0x7f0590196240 2024-02-21T23:42:21.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:21 smithi178 ceph-mon[47547]: Reconfiguring mon.c (monmap changed)... 2024-02-21T23:42:21.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:21 smithi178 ceph-mon[47547]: Reconfiguring daemon mon.c on smithi178 2024-02-21T23:42:22.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:22.088+0000 7f0594c7d700 1 -- 172.21.15.178:0/122746529 --> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi090=x;smithi176=y;smithi178=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f05900611b0 con 0x7f0578038c20 2024-02-21T23:42:22.080 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:22.095+0000 7f057f7fe700 1 -- 172.21.15.178:0/122746529 <== mgr.14150 v2:172.21.15.90:6800/3697394685 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f05900611b0 con 0x7f0578038c20 2024-02-21T23:42:22.080 INFO:teuthology.orchestra.run.smithi178.stdout:Scheduled mgr update... 2024-02-21T23:42:22.082 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:22.097+0000 7f0594c7d700 1 -- 172.21.15.178:0/122746529 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f0578038c20 msgr2=0x7f057803b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:42:22.082 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:22.097+0000 7f0594c7d700 1 --2- 172.21.15.178:0/122746529 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f0578038c20 0x7f057803b0e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f058400b5c0 tx=0x7f0584005200 comp rx=0 tx=0).stop 2024-02-21T23:42:22.082 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:22.097+0000 7f0594c7d700 1 -- 172.21.15.178:0/122746529 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0590196240 msgr2=0x7f059019b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:42:22.082 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:22.097+0000 7f0594c7d700 1 --2- 172.21.15.178:0/122746529 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0590196240 0x7f059019b250 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f058000b780 tx=0x7f058000ba90 comp rx=0 tx=0).stop 2024-02-21T23:42:22.083 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:22.097+0000 7f0594c7d700 1 -- 172.21.15.178:0/122746529 shutdown_connections 2024-02-21T23:42:22.083 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:22.097+0000 7f0594c7d700 1 --2- 172.21.15.178:0/122746529 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f0578038c20 0x7f057803b0e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:22.083 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:22.097+0000 7f0594c7d700 1 --2- 172.21.15.178:0/122746529 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05901966c0 0x7f059019d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:22.083 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:22.097+0000 7f0594c7d700 1 --2- 172.21.15.178:0/122746529 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f05901007e0 0x7f0590195d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:22.083 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:22.097+0000 7f0594c7d700 1 --2- 172.21.15.178:0/122746529 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0590196240 0x7f059019b250 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:22.083 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:22.097+0000 7f0594c7d700 1 -- 172.21.15.178:0/122746529 >> 172.21.15.178:0/122746529 conn(0x7f05900fbda0 msgr2=0x7f05900fca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:42:22.083 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:22.097+0000 7f0594c7d700 1 -- 172.21.15.178:0/122746529 shutdown_connections 2024-02-21T23:42:22.083 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:42:22.097+0000 7f0594c7d700 1 -- 172.21.15.178:0/122746529 wait complete. 2024-02-21T23:42:22.535 DEBUG:teuthology.orchestra.run.smithi176:mgr.y> sudo journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@mgr.y.service 2024-02-21T23:42:22.538 DEBUG:teuthology.orchestra.run.smithi178:mgr.z> sudo journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@mgr.z.service 2024-02-21T23:42:22.540 INFO:tasks.cephadm:Deploying OSDs... 2024-02-21T23:42:22.540 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-21T23:42:22.540 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/scratch_devs of=/dev/stdout 2024-02-21T23:42:22.561 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-21T23:42:22.561 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_1 2024-02-21T23:42:22.570 INFO:journalctl@ceph.mgr.y.smithi176.stdout:-- Logs begin at Wed 2024-02-21 23:23:58 UTC. -- 2024-02-21T23:42:22.571 INFO:journalctl@ceph.mgr.z.smithi178.stdout:-- Logs begin at Wed 2024-02-21 23:24:26 UTC. -- 2024-02-21T23:42:22.621 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-21T23:42:22.621 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-21T23:42:22.622 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 6h/6d Inode: 176726 Links: 1 2024-02-21T23:42:22.622 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-21T23:42:22.622 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-02-21T23:42:22.622 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-02-21 23:41:42.656270581 +0000 2024-02-21T23:42:22.622 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-02-21 23:33:59.553714384 +0000 2024-02-21T23:42:22.622 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-02-21 23:33:59.553714384 +0000 2024-02-21T23:42:22.622 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: - 2024-02-21T23:42:22.622 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-21T23:42:22.690 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-02-21T23:42:22.691 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-02-21T23:42:22.691 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000263174 s, 1.9 MB/s 2024-02-21T23:42:22.692 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-21T23:42:22.725 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:22 smithi178 ceph-mon[47547]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:22.725 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:22 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:22.725 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:22 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:22.725 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:22 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:42:22.725 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:22 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:22.726 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:22 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:42:22.726 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:22 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:22.726 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:22 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:22.726 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:22 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:42:22.726 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:22 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:22.726 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:22 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:42:22.726 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:22 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:22.726 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:22 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-21T23:42:22.726 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:22 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-21T23:42:22.726 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:22 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-21T23:42:22.726 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:22 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:22.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:22 smithi090 ceph-mon[43385]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:22.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:22 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:22.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:22 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:22.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:22 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:42:22.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:22 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:22.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:22 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:42:22.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:22 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:22.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:22 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:22.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:22 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:42:22.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:22 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:22.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:22 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:42:22.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:22 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:22.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:22 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-21T23:42:22.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:22 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-21T23:42:22.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:22 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-21T23:42:22.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:22 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:22.748 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_2 2024-02-21T23:42:22.807 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-21T23:42:22.807 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-21T23:42:22.807 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 6h/6d Inode: 179832 Links: 1 2024-02-21T23:42:22.807 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-21T23:42:22.807 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-02-21T23:42:22.807 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-02-21 23:41:42.714269649 +0000 2024-02-21T23:42:22.808 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-02-21 23:33:59.931708394 +0000 2024-02-21T23:42:22.808 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-02-21 23:33:59.931708394 +0000 2024-02-21T23:42:22.808 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: - 2024-02-21T23:42:22.808 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-21T23:42:22.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:22 smithi176 ceph-mon[47172]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:22.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:22 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:22.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:22 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:22.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:22 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:42:22.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:22 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:22.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:22 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:42:22.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:22 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:22.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:22 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:22.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:22 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:42:22.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:22 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:22.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:22 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:42:22.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:22 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:22.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:22 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-21T23:42:22.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:22 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-21T23:42:22.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:22 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-21T23:42:22.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:22 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:22.875 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-02-21T23:42:22.875 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-02-21T23:42:22.875 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000251973 s, 2.0 MB/s 2024-02-21T23:42:22.876 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-21T23:42:22.935 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_3 2024-02-21T23:42:22.993 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-21T23:42:22.993 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-21T23:42:22.993 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 6h/6d Inode: 176805 Links: 1 2024-02-21T23:42:22.993 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-21T23:42:22.993 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-02-21T23:42:22.993 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-02-21 23:41:42.772268718 +0000 2024-02-21T23:42:22.993 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-02-21 23:34:00.282702831 +0000 2024-02-21T23:42:22.994 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-02-21 23:34:00.282702831 +0000 2024-02-21T23:42:22.994 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: - 2024-02-21T23:42:22.994 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-21T23:42:23.060 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-02-21T23:42:23.060 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-02-21T23:42:23.060 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000190178 s, 2.7 MB/s 2024-02-21T23:42:23.061 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-21T23:42:23.120 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_4 2024-02-21T23:42:23.177 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-21T23:42:23.177 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-21T23:42:23.177 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 6h/6d Inode: 174855 Links: 1 2024-02-21T23:42:23.177 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-21T23:42:23.177 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-02-21T23:42:23.177 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-02-21 23:41:42.828267819 +0000 2024-02-21T23:42:23.177 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-02-21 23:34:00.641697142 +0000 2024-02-21T23:42:23.177 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-02-21 23:34:00.641697142 +0000 2024-02-21T23:42:23.177 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: - 2024-02-21T23:42:23.178 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-21T23:42:23.243 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-02-21T23:42:23.243 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-02-21T23:42:23.243 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000217013 s, 2.4 MB/s 2024-02-21T23:42:23.244 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-21T23:42:23.302 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-02-21T23:42:23.302 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/scratch_devs of=/dev/stdout 2024-02-21T23:42:23.323 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-21T23:42:23.323 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_1 2024-02-21T23:42:23.383 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-21T23:42:23.384 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-21T23:42:23.384 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 6h/6d Inode: 178592 Links: 1 2024-02-21T23:42:23.384 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-21T23:42:23.384 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-02-21T23:42:23.384 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-02-21 23:42:11.149005045 +0000 2024-02-21T23:42:23.384 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-02-21 23:34:19.783089619 +0000 2024-02-21T23:42:23.384 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-02-21 23:34:19.783089619 +0000 2024-02-21T23:42:23.384 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: - 2024-02-21T23:42:23.384 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-21T23:42:23.454 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-02-21T23:42:23.455 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-02-21T23:42:23.455 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000241152 s, 2.1 MB/s 2024-02-21T23:42:23.456 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-21T23:42:23.517 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_2 2024-02-21T23:42:23.576 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-21T23:42:23.576 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-21T23:42:23.576 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 6h/6d Inode: 175865 Links: 1 2024-02-21T23:42:23.576 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-21T23:42:23.576 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-02-21T23:42:23.576 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-02-21 23:42:11.149005045 +0000 2024-02-21T23:42:23.576 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-02-21 23:34:20.120086551 +0000 2024-02-21T23:42:23.576 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-02-21 23:34:20.120086551 +0000 2024-02-21T23:42:23.576 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: - 2024-02-21T23:42:23.577 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-21T23:42:23.644 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-02-21T23:42:23.644 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-02-21T23:42:23.644 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000233224 s, 2.2 MB/s 2024-02-21T23:42:23.645 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-21T23:42:23.706 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_3 2024-02-21T23:42:23.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:23 smithi090 ceph-mon[43385]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi090=x;smithi176=y;smithi178=z", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:42:23.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:23 smithi090 ceph-mon[43385]: Saving service mgr spec with placement smithi090=x;smithi176=y;smithi178=z;count:3 2024-02-21T23:42:23.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:23 smithi090 ceph-mon[43385]: Deploying daemon mgr.z on smithi178 2024-02-21T23:42:23.763 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Feb 21 23:42:23 smithi178 systemd[1]: Starting Ceph mgr.z for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:42:23.763 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:23 smithi178 ceph-mon[47547]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi090=x;smithi176=y;smithi178=z", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:42:23.763 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:23 smithi178 ceph-mon[47547]: Saving service mgr spec with placement smithi090=x;smithi176=y;smithi178=z;count:3 2024-02-21T23:42:23.763 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:23 smithi178 ceph-mon[47547]: Deploying daemon mgr.z on smithi178 2024-02-21T23:42:23.765 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-21T23:42:23.765 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-21T23:42:23.765 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 6h/6d Inode: 180646 Links: 1 2024-02-21T23:42:23.765 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-21T23:42:23.765 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-02-21T23:42:23.765 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-02-21 23:42:11.149005045 +0000 2024-02-21T23:42:23.765 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-02-21 23:34:20.448083566 +0000 2024-02-21T23:42:23.765 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-02-21 23:34:20.448083566 +0000 2024-02-21T23:42:23.765 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: - 2024-02-21T23:42:23.766 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-21T23:42:23.830 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:23 smithi176 ceph-mon[47172]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi090=x;smithi176=y;smithi178=z", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:42:23.830 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:23 smithi176 ceph-mon[47172]: Saving service mgr spec with placement smithi090=x;smithi176=y;smithi178=z;count:3 2024-02-21T23:42:23.830 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:23 smithi176 ceph-mon[47172]: Deploying daemon mgr.z on smithi178 2024-02-21T23:42:23.832 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-02-21T23:42:23.832 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-02-21T23:42:23.832 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000262285 s, 2.0 MB/s 2024-02-21T23:42:23.834 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-21T23:42:23.894 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_4 2024-02-21T23:42:23.952 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-21T23:42:23.952 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-21T23:42:23.952 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 6h/6d Inode: 176023 Links: 1 2024-02-21T23:42:23.952 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-21T23:42:23.952 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-02-21T23:42:23.953 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-02-21 23:42:11.149005045 +0000 2024-02-21T23:42:23.953 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-02-21 23:34:20.813080243 +0000 2024-02-21T23:42:23.953 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-02-21 23:34:20.813080243 +0000 2024-02-21T23:42:23.953 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: - 2024-02-21T23:42:23.953 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-21T23:42:24.018 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-02-21T23:42:24.018 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-02-21T23:42:24.018 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000236548 s, 2.2 MB/s 2024-02-21T23:42:24.019 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-21T23:42:24.080 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-02-21T23:42:24.080 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/scratch_devs of=/dev/stdout 2024-02-21T23:42:24.098 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-21T23:42:24.098 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_1 2024-02-21T23:42:24.156 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-21T23:42:24.156 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-21T23:42:24.156 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 6h/6d Inode: 177802 Links: 1 2024-02-21T23:42:24.156 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-21T23:42:24.156 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2024-02-21T23:42:24.156 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-02-21 23:42:23.459854996 +0000 2024-02-21T23:42:24.156 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-02-21 23:34:25.789962332 +0000 2024-02-21T23:42:24.156 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-02-21 23:34:25.789962332 +0000 2024-02-21T23:42:24.156 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: - 2024-02-21T23:42:24.157 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-21T23:42:24.174 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Feb 21 23:42:23 smithi178 ceph-mgr[49419]: -- 172.21.15.178:0/3403483257 <== mon.1 v2:172.21.15.178:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e34624f1e0 con 0x55e346224800 2024-02-21T23:42:24.174 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Feb 21 23:42:23 smithi178 bash[49382]: a555e674b741974a90567bd2714c86d1f9f18e5d23f75217f076747e2599e4c8 2024-02-21T23:42:24.174 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Feb 21 23:42:23 smithi178 systemd[1]: Started Ceph mgr.z for 7ed81218-d112-11ee-95bd-87774f69a715. 2024-02-21T23:42:24.202 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-02-21T23:42:24.202 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-02-21T23:42:24.202 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 9.0982e-05 s, 5.6 MB/s 2024-02-21T23:42:24.203 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-21T23:42:24.261 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_2 2024-02-21T23:42:24.319 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-21T23:42:24.319 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-21T23:42:24.319 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 6h/6d Inode: 180375 Links: 1 2024-02-21T23:42:24.319 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-21T23:42:24.319 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2024-02-21T23:42:24.319 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-02-21 23:42:23.459854996 +0000 2024-02-21T23:42:24.319 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-02-21 23:34:26.129956183 +0000 2024-02-21T23:42:24.319 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-02-21 23:34:26.129956183 +0000 2024-02-21T23:42:24.319 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: - 2024-02-21T23:42:24.320 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-21T23:42:24.383 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Feb 21 23:42:24 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-z[49397]: 2024-02-21T23:42:24.308+0000 7f5998aa6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-21T23:42:24.385 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-02-21T23:42:24.385 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-02-21T23:42:24.385 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 0.000100298 s, 5.1 MB/s 2024-02-21T23:42:24.386 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-21T23:42:24.444 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_3 2024-02-21T23:42:24.464 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Feb 21 23:42:24 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-z[49397]: 2024-02-21T23:42:24.407+0000 7f5998aa6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-21T23:42:24.480 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-21T23:42:24.480 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-21T23:42:24.480 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 6h/6d Inode: 182275 Links: 1 2024-02-21T23:42:24.481 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-21T23:42:24.481 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2024-02-21T23:42:24.481 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-02-21 23:42:23.459854996 +0000 2024-02-21T23:42:24.481 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-02-21 23:34:26.456950270 +0000 2024-02-21T23:42:24.481 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-02-21 23:34:26.456950270 +0000 2024-02-21T23:42:24.481 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: - 2024-02-21T23:42:24.481 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-21T23:42:24.547 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-02-21T23:42:24.547 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-02-21T23:42:24.547 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 9.0258e-05 s, 5.7 MB/s 2024-02-21T23:42:24.548 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-21T23:42:24.607 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_4 2024-02-21T23:42:24.665 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-21T23:42:24.665 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-21T23:42:24.665 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 6h/6d Inode: 180496 Links: 1 2024-02-21T23:42:24.665 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-21T23:42:24.665 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2024-02-21T23:42:24.666 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-02-21 23:42:23.459854996 +0000 2024-02-21T23:42:24.666 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-02-21 23:34:26.785944320 +0000 2024-02-21T23:42:24.666 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-02-21 23:34:26.785944320 +0000 2024-02-21T23:42:24.666 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: - 2024-02-21T23:42:24.666 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-21T23:42:24.674 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Feb 21 23:42:24 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-z[49397]: 2024-02-21T23:42:24.501+0000 7f5998aa6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-21T23:42:24.674 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Feb 21 23:42:24 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-z[49397]: 2024-02-21T23:42:24.592+0000 7f5998aa6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-21T23:42:24.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:24 smithi178 ceph-mon[47547]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:24.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:24 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:24.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:24 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:24.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:24 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:24.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:24 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-21T23:42:24.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:24 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-21T23:42:24.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:24 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-21T23:42:24.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:24 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:24.675 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:24 smithi176 ceph-mon[47172]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:24.675 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:24 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:24.676 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:24 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:24.676 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:24 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:24.676 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:24 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-21T23:42:24.676 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:24 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-21T23:42:24.676 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:24 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-21T23:42:24.676 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:24 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:24.700 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-02-21T23:42:24.700 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-02-21T23:42:24.700 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 9.1427e-05 s, 5.6 MB/s 2024-02-21T23:42:24.701 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-21T23:42:24.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:24 smithi090 ceph-mon[43385]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:24.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:24 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:24.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:24 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:24.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:24 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:24.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:24 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-21T23:42:24.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:24 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-21T23:42:24.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:24 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-21T23:42:24.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:24 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:24.759 INFO:tasks.cephadm:Deploying osd.0 on smithi090 with /dev/vg_nvme/lv_4... 2024-02-21T23:42:24.759 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-21T23:42:24.985 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.a/config 2024-02-21T23:42:25.334 INFO:journalctl@ceph.mgr.y.smithi176.stdout:Feb 21 23:42:25 smithi176 systemd[1]: Starting Ceph mgr.y for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:42:25.483 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:25 smithi178 ceph-mon[47547]: Deploying daemon mgr.y on smithi176 2024-02-21T23:42:25.484 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Feb 21 23:42:25 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-z[49397]: 2024-02-21T23:42:25.244+0000 7f5998aa6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-21T23:42:25.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:25 smithi090 ceph-mon[43385]: Deploying daemon mgr.y on smithi176 2024-02-21T23:42:25.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:25 smithi176 ceph-mon[47172]: Deploying daemon mgr.y on smithi176 2024-02-21T23:42:25.834 INFO:journalctl@ceph.mgr.y.smithi176.stdout:Feb 21 23:42:25 smithi176 ceph-mgr[48675]: -- 172.21.15.176:0/2927600993 <== mon.2 v2:172.21.15.176:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55adcca251e0 con 0x55adcc9fb000 2024-02-21T23:42:25.834 INFO:journalctl@ceph.mgr.y.smithi176.stdout:Feb 21 23:42:25 smithi176 bash[48637]: 9b255cb3db1552c64af9f2c067eb778c309744a7cfa3d90e0427f2f6393dd988 2024-02-21T23:42:25.834 INFO:journalctl@ceph.mgr.y.smithi176.stdout:Feb 21 23:42:25 smithi176 systemd[1]: Started Ceph mgr.y for 7ed81218-d112-11ee-95bd-87774f69a715. 2024-02-21T23:42:25.837 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Feb 21 23:42:25 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-z[49397]: 2024-02-21T23:42:25.498+0000 7f5998aa6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-21T23:42:25.837 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Feb 21 23:42:25 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-z[49397]: 2024-02-21T23:42:25.583+0000 7f5998aa6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-21T23:42:25.838 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Feb 21 23:42:25 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-z[49397]: 2024-02-21T23:42:25.692+0000 7f5998aa6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-21T23:42:25.838 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Feb 21 23:42:25 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-z[49397]: 2024-02-21T23:42:25.852+0000 7f5998aa6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-21T23:42:26.102 INFO:journalctl@ceph.mgr.y.smithi176.stdout:Feb 21 23:42:26 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-y[48652]: 2024-02-21T23:42:26.013+0000 7fa73a5ad200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-21T23:42:26.174 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Feb 21 23:42:25 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-z[49397]: 2024-02-21T23:42:25.985+0000 7f5998aa6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-21T23:42:26.174 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Feb 21 23:42:26 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-z[49397]: 2024-02-21T23:42:26.079+0000 7f5998aa6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-21T23:42:26.278 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-21T23:42:26.278 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-21T23:42:26.278 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10+0 records in 2024-02-21T23:42:26.279 INFO:teuthology.orchestra.run.smithi090.stderr:10+0 records out 2024-02-21T23:42:26.279 INFO:teuthology.orchestra.run.smithi090.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113038 s, 928 MB/s 2024-02-21T23:42:26.279 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping successful for: 2024-02-21T23:42:26.290 INFO:journalctl@ceph.mgr.y.smithi176.stdout:Feb 21 23:42:26 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-y[48652]: 2024-02-21T23:42:26.117+0000 7fa73a5ad200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-21T23:42:26.290 INFO:journalctl@ceph.mgr.y.smithi176.stdout:Feb 21 23:42:26 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-y[48652]: 2024-02-21T23:42:26.217+0000 7fa73a5ad200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-21T23:42:26.470 INFO:journalctl@ceph.mgr.y.smithi176.stdout:Feb 21 23:42:26 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-y[48652]: 2024-02-21T23:42:26.305+0000 7fa73a5ad200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-21T23:42:26.660 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:26 smithi090 ceph-mon[43385]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:26.660 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:26 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:26.661 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:26 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:26.661 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:26 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:26.661 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:26 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:26.661 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:26 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:42:26.662 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph orch daemon add osd smithi090:vg_nvme/lv_4 2024-02-21T23:42:26.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:26 smithi176 ceph-mon[47172]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:26.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:26 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:26.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:26 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:26.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:26 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:26.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:26 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:26.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:26 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:42:26.879 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Feb 21 23:42:26 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-z[49397]: 2024-02-21T23:42:26.493+0000 7f5998aa6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-21T23:42:26.880 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:26 smithi178 ceph-mon[47547]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:26.880 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:26 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:26.880 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:26 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:26.880 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:26 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:26.880 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:26 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:26.880 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:26 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:42:26.882 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.a/config 2024-02-21T23:42:27.163 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Feb 21 23:42:26 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-z[49397]: 2024-02-21T23:42:26.894+0000 7f5998aa6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-21T23:42:27.163 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Feb 21 23:42:27 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-z[49397]: 2024-02-21T23:42:27.085+0000 7f5998aa6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-21T23:42:27.163 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Feb 21 23:42:27 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-z[49397]: 2024-02-21T23:42:27.178+0000 7f5998aa6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-21T23:42:27.289 INFO:journalctl@ceph.mgr.y.smithi176.stdout:Feb 21 23:42:26 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-y[48652]: 2024-02-21T23:42:26.970+0000 7fa73a5ad200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-21T23:42:27.289 INFO:journalctl@ceph.mgr.y.smithi176.stdout:Feb 21 23:42:27 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-y[48652]: 2024-02-21T23:42:27.218+0000 7fa73a5ad200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-21T23:42:27.551 INFO:journalctl@ceph.mgr.y.smithi176.stdout:Feb 21 23:42:27 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-y[48652]: 2024-02-21T23:42:27.304+0000 7fa73a5ad200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-21T23:42:27.551 INFO:journalctl@ceph.mgr.y.smithi176.stdout:Feb 21 23:42:27 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-y[48652]: 2024-02-21T23:42:27.411+0000 7fa73a5ad200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-21T23:42:27.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:27 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:27.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:27 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:27.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:27 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:27.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:27 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:27.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:27 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:27.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:27 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:42:27.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:27 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:27.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:27 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-21T23:42:27.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:27 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-21T23:42:27.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:27 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:27.834 INFO:journalctl@ceph.mgr.y.smithi176.stdout:Feb 21 23:42:27 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-y[48652]: 2024-02-21T23:42:27.569+0000 7fa73a5ad200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-21T23:42:27.834 INFO:journalctl@ceph.mgr.y.smithi176.stdout:Feb 21 23:42:27 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-y[48652]: 2024-02-21T23:42:27.694+0000 7fa73a5ad200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-21T23:42:27.834 INFO:journalctl@ceph.mgr.y.smithi176.stdout:Feb 21 23:42:27 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-y[48652]: 2024-02-21T23:42:27.778+0000 7fa73a5ad200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-21T23:42:27.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:27 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:27.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:27 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:27.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:27 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:27.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:27 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:27.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:27 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:27.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:27 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:42:27.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:27 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:27.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:27 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-21T23:42:27.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:27 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-21T23:42:27.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:27 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:27.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:27 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:27.919 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:27 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:27.919 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:27 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:27.919 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:27 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:27.919 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:27 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:27.919 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:27 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:42:27.919 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:27 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:27.919 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:27 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-21T23:42:27.919 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:27 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-21T23:42:27.919 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:27 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:28.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.180+0000 7f5792433700 1 -- 172.21.15.90:0/3229202383 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f578c0ffc90 msgr2=0x7f578c1000b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:42:28.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.180+0000 7f5792433700 1 --2- 172.21.15.90:0/3229202383 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f578c0ffc90 0x7f578c1000b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f577c0099c0 tx=0x7f577c009cd0 comp rx=0 tx=0).stop 2024-02-21T23:42:28.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.180+0000 7f5792433700 1 -- 172.21.15.90:0/3229202383 shutdown_connections 2024-02-21T23:42:28.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.180+0000 7f5792433700 1 --2- 172.21.15.90:0/3229202383 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f578c101830 0x7f578c1080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:28.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.180+0000 7f5792433700 1 --2- 172.21.15.90:0/3229202383 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f578c0ffc90 0x7f578c1000b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:28.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.180+0000 7f5792433700 1 --2- 172.21.15.90:0/3229202383 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f578c100e90 0x7f578c1012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:28.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.180+0000 7f5792433700 1 -- 172.21.15.90:0/3229202383 >> 172.21.15.90:0/3229202383 conn(0x7f578c0fb230 msgr2=0x7f578c0fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:42:28.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.180+0000 7f5792433700 1 -- 172.21.15.90:0/3229202383 shutdown_connections 2024-02-21T23:42:28.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.180+0000 7f5792433700 1 -- 172.21.15.90:0/3229202383 wait complete. 2024-02-21T23:42:28.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.181+0000 7f5792433700 1 Processor -- start 2024-02-21T23:42:28.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.181+0000 7f5792433700 1 -- start start 2024-02-21T23:42:28.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.181+0000 7f5792433700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f578c0ffc90 0x7f578c069ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:42:28.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.181+0000 7f5792433700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f578c100e90 0x7f578c19ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:42:28.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.181+0000 7f5792433700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f578c101830 0x7f578c19e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:42:28.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.181+0000 7f5792433700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578c19e800 con 0x7f578c101830 2024-02-21T23:42:28.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.181+0000 7f5792433700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578c19e970 con 0x7f578c0ffc90 2024-02-21T23:42:28.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.181+0000 7f5792433700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578c19eb30 con 0x7f578c100e90 2024-02-21T23:42:28.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.181+0000 7f57909d0700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f578c101830 0x7f578c19e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:42:28.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.181+0000 7f57909d0700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f578c101830 0x7f578c19e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60854/0 (socket says 172.21.15.90:60854) 2024-02-21T23:42:28.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.181+0000 7f57909d0700 1 -- 172.21.15.90:0/3186073034 learned_addr learned my addr 172.21.15.90:0/3186073034 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:42:28.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.181+0000 7f578bfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f578c0ffc90 0x7f578c069ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:42:28.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.181+0000 7f578b7fe700 1 --2- 172.21.15.90:0/3186073034 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f578c100e90 0x7f578c19ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:42:28.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.182+0000 7f57909d0700 1 -- 172.21.15.90:0/3186073034 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f578c100e90 msgr2=0x7f578c19ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:42:28.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.182+0000 7f57909d0700 1 --2- 172.21.15.90:0/3186073034 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f578c100e90 0x7f578c19ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:28.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.182+0000 7f57909d0700 1 -- 172.21.15.90:0/3186073034 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f578c0ffc90 msgr2=0x7f578c069ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:42:28.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.182+0000 7f57909d0700 1 --2- 172.21.15.90:0/3186073034 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f578c0ffc90 0x7f578c069ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:28.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.182+0000 7f57909d0700 1 -- 172.21.15.90:0/3186073034 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f577c0095f0 con 0x7f578c101830 2024-02-21T23:42:28.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.182+0000 7f57909d0700 1 --2- 172.21.15.90:0/3186073034 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f578c101830 0x7f578c19e060 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5784007ee0 tx=0x7f578400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:42:28.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.182+0000 7f57897fa700 1 -- 172.21.15.90:0/3186073034 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5784017070 con 0x7f578c101830 2024-02-21T23:42:28.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.182+0000 7f57897fa700 1 -- 172.21.15.90:0/3186073034 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f578400f040 con 0x7f578c101830 2024-02-21T23:42:28.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.182+0000 7f5792433700 1 -- 172.21.15.90:0/3186073034 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f578c19ee10 con 0x7f578c101830 2024-02-21T23:42:28.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.182+0000 7f57897fa700 1 -- 172.21.15.90:0/3186073034 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5784005580 con 0x7f578c101830 2024-02-21T23:42:28.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.182+0000 7f5792433700 1 -- 172.21.15.90:0/3186073034 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f578c19f280 con 0x7f578c101830 2024-02-21T23:42:28.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.183+0000 7f57897fa700 1 -- 172.21.15.90:0/3186073034 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) v1 ==== 45593+0+0 (secure 0 0 0) 0x7f57840056e0 con 0x7f578c101830 2024-02-21T23:42:28.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.183+0000 7f5792433700 1 -- 172.21.15.90:0/3186073034 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f578c1071b0 con 0x7f578c101830 2024-02-21T23:42:28.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.183+0000 7f57897fa700 1 --2- 172.21.15.90:0/3186073034 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f57740389d0 0x7f577403ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:42:28.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.183+0000 7f57897fa700 1 -- 172.21.15.90:0/3186073034 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f578404a940 con 0x7f578c101830 2024-02-21T23:42:28.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.183+0000 7f578bfff700 1 --2- 172.21.15.90:0/3186073034 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f57740389d0 0x7f577403ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:42:28.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.183+0000 7f578bfff700 1 --2- 172.21.15.90:0/3186073034 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f57740389d0 0x7f577403ae90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f577c006010 tx=0x7f577c00b540 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:42:28.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.188+0000 7f57897fa700 1 -- 172.21.15.90:0/3186073034 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f57840112b0 con 0x7f578c101830 2024-02-21T23:42:28.174 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Feb 21 23:42:27 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-z[49397]: 2024-02-21T23:42:27.933+0000 7f5998aa6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-21T23:42:28.174 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Feb 21 23:42:28 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-z[49397]: 2024-02-21T23:42:28.028+0000 7f5998aa6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-21T23:42:28.174 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Feb 21 23:42:28 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-z[49397]: 2024-02-21T23:42:28.178+0000 7f5998aa6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-21T23:42:28.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:28.391+0000 7f5792433700 1 -- 172.21.15.90:0/3186073034 --> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f578c0611b0 con 0x7f57740389d0 2024-02-21T23:42:28.523 INFO:journalctl@ceph.mgr.y.smithi176.stdout:Feb 21 23:42:28 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-y[48652]: 2024-02-21T23:42:28.152+0000 7fa73a5ad200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-21T23:42:28.807 INFO:journalctl@ceph.mgr.y.smithi176.stdout:Feb 21 23:42:28 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-y[48652]: 2024-02-21T23:42:28.538+0000 7fa73a5ad200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-21T23:42:28.807 INFO:journalctl@ceph.mgr.y.smithi176.stdout:Feb 21 23:42:28 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-y[48652]: 2024-02-21T23:42:28.729+0000 7fa73a5ad200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-21T23:42:28.808 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:28 smithi176 ceph-mon[47172]: Reconfiguring mgr.x (unknown last config time)... 2024-02-21T23:42:28.808 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:28 smithi176 ceph-mon[47172]: Reconfiguring daemon mgr.x on smithi090 2024-02-21T23:42:28.808 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:28 smithi176 ceph-mon[47172]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:28.808 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:28 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:28.808 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:28 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:28.808 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:28 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:42:28.808 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:28 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:28.808 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:28 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:42:28.808 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:28 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:28.808 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:28 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-21T23:42:28.808 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:28 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-21T23:42:28.808 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:28 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:28.809 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:28 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:28.865 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:28 smithi090 ceph-mon[43385]: Reconfiguring mgr.x (unknown last config time)... 2024-02-21T23:42:28.865 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:28 smithi090 ceph-mon[43385]: Reconfiguring daemon mgr.x on smithi090 2024-02-21T23:42:28.865 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:28 smithi090 ceph-mon[43385]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:28.865 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:28 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:28.865 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:28 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:28.865 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:28 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:42:28.865 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:28 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:28.865 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:28 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:42:28.865 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:28 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:28.865 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:28 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-21T23:42:28.866 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:28 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-21T23:42:28.866 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:28 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:28.866 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:28 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:28.924 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Feb 21 23:42:28 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-z[49397]: 2024-02-21T23:42:28.806+0000 7f5998aa6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-21T23:42:28.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:28 smithi178 ceph-mon[47547]: Reconfiguring mgr.x (unknown last config time)... 2024-02-21T23:42:28.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:28 smithi178 ceph-mon[47547]: Reconfiguring daemon mgr.x on smithi090 2024-02-21T23:42:28.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:28 smithi178 ceph-mon[47547]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:28.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:28 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:28.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:28 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:28.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:28 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:42:28.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:28 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:28.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:28 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:42:28.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:28 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:28.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:28 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-21T23:42:28.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:28 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-21T23:42:28.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:28 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:28.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:28 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:29.084 INFO:journalctl@ceph.mgr.y.smithi176.stdout:Feb 21 23:42:28 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-y[48652]: 2024-02-21T23:42:28.822+0000 7fa73a5ad200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-21T23:42:29.424 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Feb 21 23:42:28 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-z[49397]: 2024-02-21T23:42:28.988+0000 7f5998aa6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-21T23:42:29.424 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Feb 21 23:42:29 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-z[49397]: 2024-02-21T23:42:29.184+0000 7f5998aa6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-21T23:42:29.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:29 smithi090 ceph-mon[43385]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:42:29.806 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:29 smithi176 ceph-mon[47172]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:42:29.807 INFO:journalctl@ceph.mgr.y.smithi176.stdout:Feb 21 23:42:29 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-y[48652]: 2024-02-21T23:42:29.580+0000 7fa73a5ad200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-21T23:42:29.807 INFO:journalctl@ceph.mgr.y.smithi176.stdout:Feb 21 23:42:29 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-y[48652]: 2024-02-21T23:42:29.676+0000 7fa73a5ad200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-21T23:42:29.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:29 smithi178 ceph-mon[47547]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:42:30.084 INFO:journalctl@ceph.mgr.y.smithi176.stdout:Feb 21 23:42:29 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-y[48652]: 2024-02-21T23:42:29.821+0000 7fa73a5ad200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-21T23:42:30.387 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Feb 21 23:42:30 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-z[49397]: 2024-02-21T23:42:30.120+0000 7f5998aa6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-21T23:42:30.388 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Feb 21 23:42:30 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-z[49397]: 2024-02-21T23:42:30.402+0000 7f5998aa6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-21T23:42:30.665 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Feb 21 23:42:30 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-z[49397]: 2024-02-21T23:42:30.489+0000 7f5998aa6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-21T23:42:30.666 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:30 smithi178 ceph-mon[47547]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:30.666 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:30 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/3191302789' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77878890-d9a5-4d5b-a943-fec29ee0446e"}]: dispatch 2024-02-21T23:42:30.666 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:30 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/3191302789' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "77878890-d9a5-4d5b-a943-fec29ee0446e"}]': finished 2024-02-21T23:42:30.666 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:30 smithi178 ceph-mon[47547]: osdmap e5: 1 total, 0 up, 1 in 2024-02-21T23:42:30.666 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:30 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-21T23:42:30.801 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:30 smithi176 ceph-mon[47172]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:30.801 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:30 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/3191302789' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77878890-d9a5-4d5b-a943-fec29ee0446e"}]: dispatch 2024-02-21T23:42:30.801 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:30 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/3191302789' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "77878890-d9a5-4d5b-a943-fec29ee0446e"}]': finished 2024-02-21T23:42:30.801 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:30 smithi176 ceph-mon[47172]: osdmap e5: 1 total, 0 up, 1 in 2024-02-21T23:42:30.801 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:30 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-21T23:42:30.802 INFO:journalctl@ceph.mgr.y.smithi176.stdout:Feb 21 23:42:30 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-y[48652]: 2024-02-21T23:42:30.445+0000 7fa73a5ad200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-21T23:42:30.802 INFO:journalctl@ceph.mgr.y.smithi176.stdout:Feb 21 23:42:30 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-y[48652]: 2024-02-21T23:42:30.625+0000 7fa73a5ad200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-21T23:42:30.802 INFO:journalctl@ceph.mgr.y.smithi176.stdout:Feb 21 23:42:30 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-y[48652]: 2024-02-21T23:42:30.816+0000 7fa73a5ad200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-21T23:42:30.924 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Feb 21 23:42:30 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-z[49397]: 2024-02-21T23:42:30.680+0000 7f5998aa6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-21T23:42:30.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:30 smithi090 ceph-mon[43385]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:30.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:30 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/3191302789' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77878890-d9a5-4d5b-a943-fec29ee0446e"}]: dispatch 2024-02-21T23:42:30.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:30 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/3191302789' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "77878890-d9a5-4d5b-a943-fec29ee0446e"}]': finished 2024-02-21T23:42:30.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:30 smithi090 ceph-mon[43385]: osdmap e5: 1 total, 0 up, 1 in 2024-02-21T23:42:30.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:30 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-21T23:42:31.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:31.587+0000 7f57897fa700 1 -- 172.21.15.90:0/3186073034 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mgrmap(e 14) v1 ==== 90707+0+0 (secure 0 0 0) 0x7f5784039550 con 0x7f578c101830 2024-02-21T23:42:31.834 INFO:journalctl@ceph.mgr.y.smithi176.stdout:Feb 21 23:42:31 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-y[48652]: 2024-02-21T23:42:31.752+0000 7fa73a5ad200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-21T23:42:31.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:31 smithi176 ceph-mon[47172]: Standby manager daemon z started 2024-02-21T23:42:31.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:31 smithi176 ceph-mon[47172]: from='mgr.? 172.21.15.178:0/4130947244' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-02-21T23:42:31.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:31 smithi176 ceph-mon[47172]: from='mgr.? 172.21.15.178:0/4130947244' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-21T23:42:31.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:31 smithi176 ceph-mon[47172]: from='mgr.? 172.21.15.178:0/4130947244' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-02-21T23:42:31.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:31 smithi176 ceph-mon[47172]: from='mgr.? 172.21.15.178:0/4130947244' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-21T23:42:31.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:31 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/3436848698' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-21T23:42:31.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:31 smithi178 ceph-mon[47547]: Standby manager daemon z started 2024-02-21T23:42:31.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:31 smithi178 ceph-mon[47547]: from='mgr.? 172.21.15.178:0/4130947244' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-02-21T23:42:31.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:31 smithi178 ceph-mon[47547]: from='mgr.? 172.21.15.178:0/4130947244' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-21T23:42:31.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:31 smithi178 ceph-mon[47547]: from='mgr.? 172.21.15.178:0/4130947244' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-02-21T23:42:31.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:31 smithi178 ceph-mon[47547]: from='mgr.? 172.21.15.178:0/4130947244' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-21T23:42:31.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:31 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/3436848698' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-21T23:42:31.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:31 smithi090 ceph-mon[43385]: Standby manager daemon z started 2024-02-21T23:42:31.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:31 smithi090 ceph-mon[43385]: from='mgr.? 172.21.15.178:0/4130947244' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-02-21T23:42:31.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:31 smithi090 ceph-mon[43385]: from='mgr.? 172.21.15.178:0/4130947244' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-21T23:42:31.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:31 smithi090 ceph-mon[43385]: from='mgr.? 172.21.15.178:0/4130947244' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-02-21T23:42:31.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:31 smithi090 ceph-mon[43385]: from='mgr.? 172.21.15.178:0/4130947244' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-21T23:42:31.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:31 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/3436848698' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-21T23:42:32.299 INFO:journalctl@ceph.mgr.y.smithi176.stdout:Feb 21 23:42:32 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-y[48652]: 2024-02-21T23:42:32.035+0000 7fa73a5ad200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-21T23:42:32.299 INFO:journalctl@ceph.mgr.y.smithi176.stdout:Feb 21 23:42:32 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-y[48652]: 2024-02-21T23:42:32.123+0000 7fa73a5ad200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-21T23:42:32.300 INFO:journalctl@ceph.mgr.y.smithi176.stdout:Feb 21 23:42:32 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-y[48652]: 2024-02-21T23:42:32.314+0000 7fa73a5ad200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-21T23:42:32.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:32.715+0000 7f57897fa700 1 -- 172.21.15.90:0/3186073034 <== mon.0 v2:172.21.15.90:3300/0 8 ==== mgrmap(e 15) v1 ==== 135821+0+0 (secure 0 0 0) 0x7f578401a310 con 0x7f578c101830 2024-02-21T23:42:32.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:32 smithi176 ceph-mon[47172]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:32.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:32 smithi176 ceph-mon[47172]: mgrmap e14: x(active, since 58s), standbys: z 2024-02-21T23:42:32.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:32 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-02-21T23:42:32.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:32 smithi176 ceph-mon[47172]: Standby manager daemon y started 2024-02-21T23:42:32.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:32 smithi176 ceph-mon[47172]: from='mgr.? 172.21.15.176:0/1040625610' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-21T23:42:32.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:32 smithi176 ceph-mon[47172]: from='mgr.? 172.21.15.176:0/1040625610' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-21T23:42:32.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:32 smithi176 ceph-mon[47172]: from='mgr.? 172.21.15.176:0/1040625610' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-21T23:42:32.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:32 smithi176 ceph-mon[47172]: from='mgr.? 172.21.15.176:0/1040625610' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-21T23:42:32.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:32 smithi178 ceph-mon[47547]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:32.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:32 smithi178 ceph-mon[47547]: mgrmap e14: x(active, since 58s), standbys: z 2024-02-21T23:42:32.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:32 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-02-21T23:42:32.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:32 smithi178 ceph-mon[47547]: Standby manager daemon y started 2024-02-21T23:42:32.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:32 smithi178 ceph-mon[47547]: from='mgr.? 172.21.15.176:0/1040625610' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-21T23:42:32.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:32 smithi178 ceph-mon[47547]: from='mgr.? 172.21.15.176:0/1040625610' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-21T23:42:32.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:32 smithi178 ceph-mon[47547]: from='mgr.? 172.21.15.176:0/1040625610' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-21T23:42:32.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:32 smithi178 ceph-mon[47547]: from='mgr.? 172.21.15.176:0/1040625610' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-21T23:42:32.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:32 smithi090 ceph-mon[43385]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:32.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:32 smithi090 ceph-mon[43385]: mgrmap e14: x(active, since 58s), standbys: z 2024-02-21T23:42:32.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:32 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-02-21T23:42:32.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:32 smithi090 ceph-mon[43385]: Standby manager daemon y started 2024-02-21T23:42:32.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:32 smithi090 ceph-mon[43385]: from='mgr.? 172.21.15.176:0/1040625610' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-21T23:42:32.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:32 smithi090 ceph-mon[43385]: from='mgr.? 172.21.15.176:0/1040625610' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-21T23:42:32.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:32 smithi090 ceph-mon[43385]: from='mgr.? 172.21.15.176:0/1040625610' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-21T23:42:32.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:32 smithi090 ceph-mon[43385]: from='mgr.? 172.21.15.176:0/1040625610' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-21T23:42:33.964 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:33 smithi090 ceph-mon[43385]: mgrmap e15: x(active, since 59s), standbys: z, y 2024-02-21T23:42:33.964 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:33 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-21T23:42:33.964 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:33 smithi090 ceph-mon[43385]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:34.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:33 smithi176 ceph-mon[47172]: mgrmap e15: x(active, since 59s), standbys: z, y 2024-02-21T23:42:34.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:33 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-21T23:42:34.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:33 smithi176 ceph-mon[47172]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:34.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:33 smithi178 ceph-mon[47547]: mgrmap e15: x(active, since 59s), standbys: z, y 2024-02-21T23:42:34.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:33 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-21T23:42:34.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:33 smithi178 ceph-mon[47547]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:36.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:36 smithi090 ceph-mon[43385]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:36.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:36 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-21T23:42:36.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:36 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:36.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:36 smithi178 ceph-mon[47547]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:36.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:36 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-21T23:42:36.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:36 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:36.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:36 smithi176 ceph-mon[47172]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:36.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:36 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-21T23:42:36.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:36 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:37.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:37 smithi090 ceph-mon[43385]: Deploying daemon osd.0 on smithi090 2024-02-21T23:42:37.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:37 smithi178 ceph-mon[47547]: Deploying daemon osd.0 on smithi090 2024-02-21T23:42:37.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:37 smithi176 ceph-mon[47172]: Deploying daemon osd.0 on smithi090 2024-02-21T23:42:38.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:38 smithi178 ceph-mon[47547]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:38.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:38 smithi090 ceph-mon[43385]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:38.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:38 smithi176 ceph-mon[47172]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:40.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:40 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:40.585 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:40 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:40.585 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:40 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:42:40.585 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:40 smithi176 ceph-mon[47172]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:40.616 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:40 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:40.616 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:40 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:40.617 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:40 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:42:40.617 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:40 smithi090 ceph-mon[43385]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:40.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:40 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:40.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:40 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:40.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:40 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:42:40.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:40 smithi178 ceph-mon[47547]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:41.525 INFO:teuthology.orchestra.run.smithi090.stdout:Created osd(s) 0 on host 'smithi090' 2024-02-21T23:42:41.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:41.540+0000 7f57897fa700 1 -- 172.21.15.90:0/3186073034 <== mgr.14150 v2:172.21.15.90:6800/3697394685 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f578c0611b0 con 0x7f57740389d0 2024-02-21T23:42:41.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:41.542+0000 7f5792433700 1 -- 172.21.15.90:0/3186073034 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f57740389d0 msgr2=0x7f577403ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:42:41.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:41.542+0000 7f5792433700 1 --2- 172.21.15.90:0/3186073034 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f57740389d0 0x7f577403ae90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f577c006010 tx=0x7f577c00b540 comp rx=0 tx=0).stop 2024-02-21T23:42:41.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:41.542+0000 7f5792433700 1 -- 172.21.15.90:0/3186073034 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f578c101830 msgr2=0x7f578c19e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:42:41.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:41.542+0000 7f5792433700 1 --2- 172.21.15.90:0/3186073034 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f578c101830 0x7f578c19e060 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5784007ee0 tx=0x7f578400d3b0 comp rx=0 tx=0).stop 2024-02-21T23:42:41.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:41.543+0000 7f5792433700 1 -- 172.21.15.90:0/3186073034 shutdown_connections 2024-02-21T23:42:41.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:41.543+0000 7f5792433700 1 --2- 172.21.15.90:0/3186073034 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f57740389d0 0x7f577403ae90 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:41.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:41.543+0000 7f5792433700 1 --2- 172.21.15.90:0/3186073034 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f578c101830 0x7f578c19e060 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:41.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:41.543+0000 7f5792433700 1 --2- 172.21.15.90:0/3186073034 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f578c0ffc90 0x7f578c069ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:41.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:41.543+0000 7f5792433700 1 --2- 172.21.15.90:0/3186073034 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f578c100e90 0x7f578c19ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:41.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:41.543+0000 7f5792433700 1 -- 172.21.15.90:0/3186073034 >> 172.21.15.90:0/3186073034 conn(0x7f578c0fb230 msgr2=0x7f578c1042f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:42:41.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:41.543+0000 7f5792433700 1 -- 172.21.15.90:0/3186073034 shutdown_connections 2024-02-21T23:42:41.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:42:41.543+0000 7f5792433700 1 -- 172.21.15.90:0/3186073034 wait complete. 2024-02-21T23:42:41.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:41 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:41.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:41 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:41.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:41 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:41.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:41 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:42:41.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:41 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:41.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:41 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:41.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:41 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:41.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:41 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:41.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:41 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:42:41.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:41 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:41.886 DEBUG:teuthology.orchestra.run.smithi090:osd.0> sudo journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@osd.0.service 2024-02-21T23:42:41.888 INFO:tasks.cephadm:Deploying osd.1 on smithi176 with /dev/vg_nvme/lv_4... 2024-02-21T23:42:41.888 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-21T23:42:41.922 INFO:journalctl@ceph.osd.0.smithi090.stdout:-- Logs begin at Wed 2024-02-21 23:24:49 UTC. -- 2024-02-21T23:42:41.926 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:41 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:41.926 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:41 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:41.926 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:41 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:41.926 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:41 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:42:41.926 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:41 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:42.115 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.b/config 2024-02-21T23:42:42.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:42 smithi090 ceph-mon[43385]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:42.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:42 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:42.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:42 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:42.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:42 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:42:42.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:42 smithi176 ceph-mon[47172]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:42.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:42 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:42.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:42 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:42.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:42 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:42:42.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:42 smithi178 ceph-mon[47547]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:42.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:42 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:42.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:42 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:42.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:42 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:42:43.350 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-21T23:42:43.350 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-21T23:42:43.350 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10+0 records in 2024-02-21T23:42:43.350 INFO:teuthology.orchestra.run.smithi176.stderr:10+0 records out 2024-02-21T23:42:43.350 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126005 s, 832 MB/s 2024-02-21T23:42:43.350 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping successful for: 2024-02-21T23:42:43.637 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:43 smithi090 ceph-mon[43385]: from='osd.0 [v2:172.21.15.90:6802/885319826,v1:172.21.15.90:6803/885319826]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-21T23:42:43.638 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:43 smithi090 ceph-mon[43385]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-21T23:42:43.638 INFO:journalctl@ceph.osd.0.smithi090.stdout:Feb 21 23:42:43 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-0[61828]: 2024-02-21T23:42:43.278+0000 7ff875043640 -1 osd.0 0 log_to_monitors true 2024-02-21T23:42:43.641 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:43 smithi176 ceph-mon[47172]: from='osd.0 [v2:172.21.15.90:6802/885319826,v1:172.21.15.90:6803/885319826]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-21T23:42:43.641 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:43 smithi176 ceph-mon[47172]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-21T23:42:43.709 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph orch daemon add osd smithi176:vg_nvme/lv_4 2024-02-21T23:42:43.924 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.b/config 2024-02-21T23:42:43.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:43 smithi178 ceph-mon[47547]: from='osd.0 [v2:172.21.15.90:6802/885319826,v1:172.21.15.90:6803/885319826]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-21T23:42:43.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:43 smithi178 ceph-mon[47547]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-21T23:42:44.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:44 smithi176 ceph-mon[47172]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:44.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:44 smithi176 ceph-mon[47172]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-21T23:42:44.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:44 smithi176 ceph-mon[47172]: osdmap e6: 1 total, 0 up, 1 in 2024-02-21T23:42:44.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:44 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-21T23:42:44.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:44 smithi176 ceph-mon[47172]: from='osd.0 [v2:172.21.15.90:6802/885319826,v1:172.21.15.90:6803/885319826]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-21T23:42:44.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:44 smithi176 ceph-mon[47172]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-21T23:42:44.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:44 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:44.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:44 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:44.585 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:44 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:42:44.585 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:44 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:44.585 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:44 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:44.585 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:44 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:42:44.585 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:44 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:44.857 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.873+0000 7f69496eb700 1 -- 172.21.15.176:0/1301268872 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6944102920 msgr2=0x7f69441091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:42:44.857 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.873+0000 7f69496eb700 1 --2- 172.21.15.176:0/1301268872 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6944102920 0x7f69441091d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f693c00b0d0 tx=0x7f693c00b3e0 comp rx=0 tx=0).stop 2024-02-21T23:42:44.858 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.873+0000 7f69496eb700 1 -- 172.21.15.176:0/1301268872 shutdown_connections 2024-02-21T23:42:44.858 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.873+0000 7f69496eb700 1 --2- 172.21.15.176:0/1301268872 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6944102920 0x7f69441091d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:44.858 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.873+0000 7f69496eb700 1 --2- 172.21.15.176:0/1301268872 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6944100d80 0x7f69441011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:44.858 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.873+0000 7f69496eb700 1 --2- 172.21.15.176:0/1301268872 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6944101f80 0x7f69441023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:44.858 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.873+0000 7f69496eb700 1 -- 172.21.15.176:0/1301268872 >> 172.21.15.176:0/1301268872 conn(0x7f69440fc320 msgr2=0x7f69440fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:42:44.858 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.873+0000 7f69496eb700 1 -- 172.21.15.176:0/1301268872 shutdown_connections 2024-02-21T23:42:44.858 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.873+0000 7f69496eb700 1 -- 172.21.15.176:0/1301268872 wait complete. 2024-02-21T23:42:44.860 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.873+0000 7f69496eb700 1 Processor -- start 2024-02-21T23:42:44.860 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.874+0000 7f69496eb700 1 -- start start 2024-02-21T23:42:44.860 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.874+0000 7f69496eb700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6944100d80 0x7f6944074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:42:44.860 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.874+0000 7f69496eb700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6944101f80 0x7f6944073160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:42:44.860 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.874+0000 7f69496eb700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6944102920 0x7f69440736a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:42:44.860 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.874+0000 7f69496eb700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6944073d80 con 0x7f6944102920 2024-02-21T23:42:44.861 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.874+0000 7f69496eb700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6944073ef0 con 0x7f6944101f80 2024-02-21T23:42:44.861 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.874+0000 7f69496eb700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69441a0e10 con 0x7f6944100d80 2024-02-21T23:42:44.861 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.874+0000 7f69427fc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6944101f80 0x7f6944073160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:42:44.861 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.874+0000 7f69427fc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6944101f80 0x7f6944073160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:44052/0 (socket says 172.21.15.176:44052) 2024-02-21T23:42:44.861 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.874+0000 7f69427fc700 1 -- 172.21.15.176:0/2991078741 learned_addr learned my addr 172.21.15.176:0/2991078741 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-21T23:42:44.861 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.874+0000 7f6942ffd700 1 --2- 172.21.15.176:0/2991078741 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6944100d80 0x7f6944074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:42:44.861 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.874+0000 7f69437fe700 1 --2- 172.21.15.176:0/2991078741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6944102920 0x7f69440736a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:42:44.861 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.874+0000 7f69427fc700 1 -- 172.21.15.176:0/2991078741 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6944100d80 msgr2=0x7f6944074a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:42:44.861 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.874+0000 7f69427fc700 1 --2- 172.21.15.176:0/2991078741 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6944100d80 0x7f6944074a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:44.861 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.874+0000 7f69427fc700 1 -- 172.21.15.176:0/2991078741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6944102920 msgr2=0x7f69440736a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:42:44.862 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.874+0000 7f69427fc700 1 --2- 172.21.15.176:0/2991078741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6944102920 0x7f69440736a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:44.862 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.874+0000 7f69427fc700 1 -- 172.21.15.176:0/2991078741 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f693c009d00 con 0x7f6944101f80 2024-02-21T23:42:44.862 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.874+0000 7f69437fe700 1 --2- 172.21.15.176:0/2991078741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6944102920 0x7f69440736a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:42:44.862 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.874+0000 7f69427fc700 1 --2- 172.21.15.176:0/2991078741 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6944101f80 0x7f6944073160 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f693800e9a0 tx=0x7f693800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:42:44.862 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.874+0000 7f692bfff700 1 -- 172.21.15.176:0/2991078741 <== mon.2 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6938009790 con 0x7f6944101f80 2024-02-21T23:42:44.862 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.875+0000 7f692bfff700 1 -- 172.21.15.176:0/2991078741 <== mon.2 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f693800cbd0 con 0x7f6944101f80 2024-02-21T23:42:44.862 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.875+0000 7f69496eb700 1 -- 172.21.15.176:0/2991078741 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69441a1070 con 0x7f6944101f80 2024-02-21T23:42:44.862 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.875+0000 7f692bfff700 1 -- 172.21.15.176:0/2991078741 <== mon.2 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6938018680 con 0x7f6944101f80 2024-02-21T23:42:44.862 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.875+0000 7f69496eb700 1 -- 172.21.15.176:0/2991078741 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69441a1540 con 0x7f6944101f80 2024-02-21T23:42:44.862 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.876+0000 7f69496eb700 1 -- 172.21.15.176:0/2991078741 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6944108140 con 0x7f6944101f80 2024-02-21T23:42:44.863 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.877+0000 7f692bfff700 1 -- 172.21.15.176:0/2991078741 <== mon.2 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 15) v1 ==== 135821+0+0 (secure 0 0 0) 0x7f693800cd40 con 0x7f6944101f80 2024-02-21T23:42:44.863 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.877+0000 7f692bfff700 1 --2- 172.21.15.176:0/2991078741 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f692c0a1090 0x7f692c0a3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:42:44.863 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.877+0000 7f692bfff700 1 -- 172.21.15.176:0/2991078741 <== mon.2 v2:172.21.15.176:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1487+0+0 (secure 0 0 0) 0x7f6938014070 con 0x7f6944101f80 2024-02-21T23:42:44.863 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.878+0000 7f6942ffd700 1 --2- 172.21.15.176:0/2991078741 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f692c0a1090 0x7f692c0a3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:42:44.863 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.878+0000 7f6942ffd700 1 --2- 172.21.15.176:0/2991078741 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f692c0a1090 0x7f692c0a3550 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6944101de0 tx=0x7f693400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:42:44.866 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:44.882+0000 7f692bfff700 1 -- 172.21.15.176:0/2991078741 <== mon.2 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f6938052960 con 0x7f6944101f80 2024-02-21T23:42:44.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:44 smithi178 ceph-mon[47547]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:44.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:44 smithi178 ceph-mon[47547]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-21T23:42:44.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:44 smithi178 ceph-mon[47547]: osdmap e6: 1 total, 0 up, 1 in 2024-02-21T23:42:44.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:44 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-21T23:42:44.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:44 smithi178 ceph-mon[47547]: from='osd.0 [v2:172.21.15.90:6802/885319826,v1:172.21.15.90:6803/885319826]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-21T23:42:44.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:44 smithi178 ceph-mon[47547]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-21T23:42:44.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:44 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:44.926 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:44 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:44.926 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:44 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:42:44.926 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:44 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:44.926 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:44 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:44.926 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:44 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:42:44.926 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:44 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:44.977 INFO:journalctl@ceph.osd.0.smithi090.stdout:Feb 21 23:42:44 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-0[61828]: 2024-02-21T23:42:44.506+0000 7ff86a6a7700 -1 osd.0 0 waiting for initial osdmap 2024-02-21T23:42:44.978 INFO:journalctl@ceph.osd.0.smithi090.stdout:Feb 21 23:42:44 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-0[61828]: 2024-02-21T23:42:44.509+0000 7ff864c99700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-21T23:42:44.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:44 smithi090 ceph-mon[43385]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:44.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:44 smithi090 ceph-mon[43385]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-21T23:42:44.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:44 smithi090 ceph-mon[43385]: osdmap e6: 1 total, 0 up, 1 in 2024-02-21T23:42:44.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:44 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-21T23:42:44.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:44 smithi090 ceph-mon[43385]: from='osd.0 [v2:172.21.15.90:6802/885319826,v1:172.21.15.90:6803/885319826]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-21T23:42:44.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:44 smithi090 ceph-mon[43385]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-21T23:42:44.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:44 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:44.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:44 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:44.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:44 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:42:44.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:44 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:44.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:44 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:44.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:44 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:42:44.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:44 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:45.065 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:45.080+0000 7f69496eb700 1 -- 172.21.15.176:0/2991078741 --> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f69440611b0 con 0x7f692c0a1090 2024-02-21T23:42:45.762 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:45 smithi176 ceph-mon[47172]: Detected new or changed devices on smithi090 2024-02-21T23:42:45.763 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:45 smithi176 ceph-mon[47172]: Adjusting osd_memory_target on smithi090 to 17113M 2024-02-21T23:42:45.763 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:45 smithi176 ceph-mon[47172]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-02-21T23:42:45.763 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:45 smithi176 ceph-mon[47172]: osdmap e7: 1 total, 0 up, 1 in 2024-02-21T23:42:45.763 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:45 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-21T23:42:45.763 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:45 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-21T23:42:45.763 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:45 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-21T23:42:45.763 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:45 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-21T23:42:45.763 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:45 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:45.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:45 smithi178 ceph-mon[47547]: Detected new or changed devices on smithi090 2024-02-21T23:42:45.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:45 smithi178 ceph-mon[47547]: Adjusting osd_memory_target on smithi090 to 17113M 2024-02-21T23:42:45.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:45 smithi178 ceph-mon[47547]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-02-21T23:42:45.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:45 smithi178 ceph-mon[47547]: osdmap e7: 1 total, 0 up, 1 in 2024-02-21T23:42:45.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:45 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-21T23:42:45.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:45 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-21T23:42:45.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:45 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-21T23:42:45.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:45 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-21T23:42:45.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:45 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:45.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:45 smithi090 ceph-mon[43385]: Detected new or changed devices on smithi090 2024-02-21T23:42:45.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:45 smithi090 ceph-mon[43385]: Adjusting osd_memory_target on smithi090 to 17113M 2024-02-21T23:42:45.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:45 smithi090 ceph-mon[43385]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-02-21T23:42:45.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:45 smithi090 ceph-mon[43385]: osdmap e7: 1 total, 0 up, 1 in 2024-02-21T23:42:45.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:45 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-21T23:42:45.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:45 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-21T23:42:45.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:45 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-21T23:42:45.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:45 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-21T23:42:45.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:45 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:46.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:46 smithi176 ceph-mon[47172]: purged_snaps scrub starts 2024-02-21T23:42:46.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:46 smithi176 ceph-mon[47172]: purged_snaps scrub ok 2024-02-21T23:42:46.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:46 smithi176 ceph-mon[47172]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:42:46.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:46 smithi176 ceph-mon[47172]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:46.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:46 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-21T23:42:46.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:46 smithi176 ceph-mon[47172]: osd.0 [v2:172.21.15.90:6802/885319826,v1:172.21.15.90:6803/885319826] boot 2024-02-21T23:42:46.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:46 smithi176 ceph-mon[47172]: osdmap e8: 1 total, 1 up, 1 in 2024-02-21T23:42:46.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:46 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-21T23:42:46.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:46 smithi178 ceph-mon[47547]: purged_snaps scrub starts 2024-02-21T23:42:46.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:46 smithi178 ceph-mon[47547]: purged_snaps scrub ok 2024-02-21T23:42:46.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:46 smithi178 ceph-mon[47547]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:42:46.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:46 smithi178 ceph-mon[47547]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:46.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:46 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-21T23:42:46.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:46 smithi178 ceph-mon[47547]: osd.0 [v2:172.21.15.90:6802/885319826,v1:172.21.15.90:6803/885319826] boot 2024-02-21T23:42:46.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:46 smithi178 ceph-mon[47547]: osdmap e8: 1 total, 1 up, 1 in 2024-02-21T23:42:46.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:46 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-21T23:42:46.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:46 smithi090 ceph-mon[43385]: purged_snaps scrub starts 2024-02-21T23:42:46.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:46 smithi090 ceph-mon[43385]: purged_snaps scrub ok 2024-02-21T23:42:46.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:46 smithi090 ceph-mon[43385]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:42:46.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:46 smithi090 ceph-mon[43385]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-21T23:42:46.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:46 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-21T23:42:46.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:46 smithi090 ceph-mon[43385]: osd.0 [v2:172.21.15.90:6802/885319826,v1:172.21.15.90:6803/885319826] boot 2024-02-21T23:42:46.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:46 smithi090 ceph-mon[43385]: osdmap e8: 1 total, 1 up, 1 in 2024-02-21T23:42:46.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:46 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-21T23:42:47.836 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:47 smithi176 ceph-mon[47172]: osdmap e9: 1 total, 1 up, 1 in 2024-02-21T23:42:47.836 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:47 smithi176 ceph-mon[47172]: from='client.? 172.21.15.176:0/1222242703' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d6ea950-7a57-4f13-ab38-10a658912783"}]: dispatch 2024-02-21T23:42:47.836 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:47 smithi176 ceph-mon[47172]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d6ea950-7a57-4f13-ab38-10a658912783"}]: dispatch 2024-02-21T23:42:47.836 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:47 smithi176 ceph-mon[47172]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d6ea950-7a57-4f13-ab38-10a658912783"}]': finished 2024-02-21T23:42:47.836 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:47 smithi176 ceph-mon[47172]: osdmap e10: 2 total, 1 up, 2 in 2024-02-21T23:42:47.836 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:47 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-21T23:42:47.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:47 smithi178 ceph-mon[47547]: osdmap e9: 1 total, 1 up, 1 in 2024-02-21T23:42:47.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:47 smithi178 ceph-mon[47547]: from='client.? 172.21.15.176:0/1222242703' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d6ea950-7a57-4f13-ab38-10a658912783"}]: dispatch 2024-02-21T23:42:47.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:47 smithi178 ceph-mon[47547]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d6ea950-7a57-4f13-ab38-10a658912783"}]: dispatch 2024-02-21T23:42:47.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:47 smithi178 ceph-mon[47547]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d6ea950-7a57-4f13-ab38-10a658912783"}]': finished 2024-02-21T23:42:47.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:47 smithi178 ceph-mon[47547]: osdmap e10: 2 total, 1 up, 2 in 2024-02-21T23:42:47.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:47 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-21T23:42:47.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:47 smithi090 ceph-mon[43385]: osdmap e9: 1 total, 1 up, 1 in 2024-02-21T23:42:47.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:47 smithi090 ceph-mon[43385]: from='client.? 172.21.15.176:0/1222242703' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d6ea950-7a57-4f13-ab38-10a658912783"}]: dispatch 2024-02-21T23:42:47.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:47 smithi090 ceph-mon[43385]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d6ea950-7a57-4f13-ab38-10a658912783"}]: dispatch 2024-02-21T23:42:47.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:47 smithi090 ceph-mon[43385]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d6ea950-7a57-4f13-ab38-10a658912783"}]': finished 2024-02-21T23:42:47.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:47 smithi090 ceph-mon[43385]: osdmap e10: 2 total, 1 up, 2 in 2024-02-21T23:42:47.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:47 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-21T23:42:48.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:48 smithi176 ceph-mon[47172]: pgmap v37: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-21T23:42:48.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:48 smithi176 ceph-mon[47172]: from='client.? 172.21.15.176:0/3751413270' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-21T23:42:48.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:48 smithi178 ceph-mon[47547]: pgmap v37: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-21T23:42:48.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:48 smithi178 ceph-mon[47547]: from='client.? 172.21.15.176:0/3751413270' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-21T23:42:48.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:48 smithi090 ceph-mon[43385]: pgmap v37: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-21T23:42:48.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:48 smithi090 ceph-mon[43385]: from='client.? 172.21.15.176:0/3751413270' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-21T23:42:50.832 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:50 smithi176 ceph-mon[47172]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-21T23:42:50.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:50 smithi178 ceph-mon[47547]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-21T23:42:50.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:50 smithi090 ceph-mon[43385]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-21T23:42:52.624 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:52 smithi176 ceph-mon[47172]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-21T23:42:52.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:52 smithi178 ceph-mon[47547]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-21T23:42:52.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:52 smithi090 ceph-mon[43385]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-21T23:42:53.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:53 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-21T23:42:53.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:53 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:53.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:53 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-21T23:42:53.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:53 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:53.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:53 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-21T23:42:53.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:53 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:54.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:54 smithi176 ceph-mon[47172]: Deploying daemon osd.1 on smithi176 2024-02-21T23:42:54.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:54 smithi176 ceph-mon[47172]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-21T23:42:54.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:54 smithi178 ceph-mon[47547]: Deploying daemon osd.1 on smithi176 2024-02-21T23:42:54.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:54 smithi178 ceph-mon[47547]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-21T23:42:54.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:54 smithi090 ceph-mon[43385]: Deploying daemon osd.1 on smithi176 2024-02-21T23:42:54.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:54 smithi090 ceph-mon[43385]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-21T23:42:56.827 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:56 smithi176 ceph-mon[47172]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-21T23:42:56.827 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:56 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:56.827 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:56 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:56.827 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:56 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:42:56.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:56 smithi178 ceph-mon[47547]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-21T23:42:56.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:56 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:56.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:56 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:56.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:56 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:42:56.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:56 smithi090 ceph-mon[43385]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-21T23:42:56.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:56 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:56.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:56 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:56.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:56 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:42:58.424 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:58.438+0000 7f692bfff700 1 -- 172.21.15.176:0/2991078741 <== mgr.14150 v2:172.21.15.90:6800/3697394685 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f69440611b0 con 0x7f692c0a1090 2024-02-21T23:42:58.424 INFO:teuthology.orchestra.run.smithi176.stdout:Created osd(s) 1 on host 'smithi176' 2024-02-21T23:42:58.425 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:58.441+0000 7f69496eb700 1 -- 172.21.15.176:0/2991078741 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f692c0a1090 msgr2=0x7f692c0a3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:42:58.425 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:58.441+0000 7f69496eb700 1 --2- 172.21.15.176:0/2991078741 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f692c0a1090 0x7f692c0a3550 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6944101de0 tx=0x7f693400b410 comp rx=0 tx=0).stop 2024-02-21T23:42:58.425 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:58.441+0000 7f69496eb700 1 -- 172.21.15.176:0/2991078741 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6944101f80 msgr2=0x7f6944073160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:42:58.426 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:58.441+0000 7f69496eb700 1 --2- 172.21.15.176:0/2991078741 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6944101f80 0x7f6944073160 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f693800e9a0 tx=0x7f693800ecb0 comp rx=0 tx=0).stop 2024-02-21T23:42:58.426 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:58.441+0000 7f69496eb700 1 -- 172.21.15.176:0/2991078741 shutdown_connections 2024-02-21T23:42:58.426 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:58.441+0000 7f69496eb700 1 --2- 172.21.15.176:0/2991078741 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f692c0a1090 0x7f692c0a3550 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:58.426 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:58.441+0000 7f69496eb700 1 --2- 172.21.15.176:0/2991078741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6944102920 0x7f69440736a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:58.426 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:58.441+0000 7f69496eb700 1 --2- 172.21.15.176:0/2991078741 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6944100d80 0x7f6944074a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:58.426 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:58.441+0000 7f69496eb700 1 --2- 172.21.15.176:0/2991078741 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6944101f80 0x7f6944073160 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:42:58.426 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:58.441+0000 7f69496eb700 1 -- 172.21.15.176:0/2991078741 >> 172.21.15.176:0/2991078741 conn(0x7f69440fc320 msgr2=0x7f6944105280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:42:58.426 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:58.441+0000 7f69496eb700 1 -- 172.21.15.176:0/2991078741 shutdown_connections 2024-02-21T23:42:58.426 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:42:58.441+0000 7f69496eb700 1 -- 172.21.15.176:0/2991078741 wait complete. 2024-02-21T23:42:58.577 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:58 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:58.577 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:58 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:58.577 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:58 smithi176 ceph-mon[47172]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-21T23:42:58.577 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:58 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:58.577 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:58 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:42:58.578 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:58 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:58.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:58 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:58.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:58 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:58.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:58 smithi178 ceph-mon[47547]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-21T23:42:58.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:58 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:58.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:58 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:42:58.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:58 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:58.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:58 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:58.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:58 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:58.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:58 smithi090 ceph-mon[43385]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-21T23:42:58.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:58 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:42:58.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:58 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:42:58.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:58 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:58.791 DEBUG:teuthology.orchestra.run.smithi176:osd.1> sudo journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@osd.1.service 2024-02-21T23:42:58.794 INFO:tasks.cephadm:Deploying osd.2 on smithi178 with /dev/vg_nvme/lv_4... 2024-02-21T23:42:58.794 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-21T23:42:58.822 INFO:journalctl@ceph.osd.1.smithi176.stdout:-- Logs begin at Wed 2024-02-21 23:23:58 UTC. -- 2024-02-21T23:42:59.020 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.c/config 2024-02-21T23:42:59.502 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:59 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:59.502 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:59 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:59.502 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:42:59 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:42:59.557 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:59 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:59.558 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:59 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:59.558 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:42:59 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:42:59.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:59 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:59.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:59 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:42:59.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:42:59 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:43:00.447 INFO:teuthology.orchestra.run.smithi178.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-21T23:43:00.447 INFO:teuthology.orchestra.run.smithi178.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-21T23:43:00.447 INFO:teuthology.orchestra.run.smithi178.stderr: stderr: 10+0 records in 2024-02-21T23:43:00.447 INFO:teuthology.orchestra.run.smithi178.stderr:10+0 records out 2024-02-21T23:43:00.447 INFO:teuthology.orchestra.run.smithi178.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00837409 s, 1.3 GB/s 2024-02-21T23:43:00.447 INFO:teuthology.orchestra.run.smithi178.stderr:--> Zapping successful for: 2024-02-21T23:43:00.498 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:00 smithi176 ceph-mon[47172]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-21T23:43:00.498 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:00 smithi176 ceph-mon[47172]: from='osd.1 [v2:172.21.15.176:6800/4085240925,v1:172.21.15.176:6801/4085240925]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-21T23:43:00.498 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:00 smithi176 ceph-mon[47172]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-21T23:43:00.498 INFO:journalctl@ceph.osd.1.smithi176.stdout:Feb 21 23:43:00 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-1[52138]: 2024-02-21T23:43:00.202+0000 7f7a71030640 -1 osd.1 0 log_to_monitors true 2024-02-21T23:43:00.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:00 smithi178 ceph-mon[47547]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-21T23:43:00.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:00 smithi178 ceph-mon[47547]: from='osd.1 [v2:172.21.15.176:6800/4085240925,v1:172.21.15.176:6801/4085240925]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-21T23:43:00.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:00 smithi178 ceph-mon[47547]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-21T23:43:00.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:00 smithi090 ceph-mon[43385]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-21T23:43:00.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:00 smithi090 ceph-mon[43385]: from='osd.1 [v2:172.21.15.176:6800/4085240925,v1:172.21.15.176:6801/4085240925]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-21T23:43:00.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:00 smithi090 ceph-mon[43385]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-21T23:43:00.862 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph orch daemon add osd smithi178:vg_nvme/lv_4 2024-02-21T23:43:01.081 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.c/config 2024-02-21T23:43:01.673 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:01 smithi178 ceph-mon[47547]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-21T23:43:01.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:01 smithi178 ceph-mon[47547]: osdmap e11: 2 total, 1 up, 2 in 2024-02-21T23:43:01.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:01 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-21T23:43:01.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:01 smithi178 ceph-mon[47547]: from='osd.1 [v2:172.21.15.176:6800/4085240925,v1:172.21.15.176:6801/4085240925]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-21T23:43:01.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:01 smithi178 ceph-mon[47547]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-21T23:43:01.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:01 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:01.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:01 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:01.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:01 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:43:01.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:01 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:01.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:01 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:01.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:01 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:43:01.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:01 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:01.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:01 smithi090 ceph-mon[43385]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-21T23:43:01.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:01 smithi090 ceph-mon[43385]: osdmap e11: 2 total, 1 up, 2 in 2024-02-21T23:43:01.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:01 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-21T23:43:01.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:01 smithi090 ceph-mon[43385]: from='osd.1 [v2:172.21.15.176:6800/4085240925,v1:172.21.15.176:6801/4085240925]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-21T23:43:01.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:01 smithi090 ceph-mon[43385]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-21T23:43:01.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:01 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:01.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:01 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:01.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:01 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:43:01.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:01 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:01.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:01 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:01.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:01 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:43:01.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:01 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:01.834 INFO:journalctl@ceph.osd.1.smithi176.stdout:Feb 21 23:43:01 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-1[52138]: 2024-02-21T23:43:01.405+0000 7f7a66694700 -1 osd.1 0 waiting for initial osdmap 2024-02-21T23:43:01.834 INFO:journalctl@ceph.osd.1.smithi176.stdout:Feb 21 23:43:01 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-1[52138]: 2024-02-21T23:43:01.408+0000 7f7a5fc84700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-21T23:43:01.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:01 smithi176 ceph-mon[47172]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-21T23:43:01.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:01 smithi176 ceph-mon[47172]: osdmap e11: 2 total, 1 up, 2 in 2024-02-21T23:43:01.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:01 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-21T23:43:01.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:01 smithi176 ceph-mon[47172]: from='osd.1 [v2:172.21.15.176:6800/4085240925,v1:172.21.15.176:6801/4085240925]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-21T23:43:01.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:01 smithi176 ceph-mon[47172]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-21T23:43:01.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:01 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:01.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:01 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:01.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:01 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:43:01.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:01 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:01.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:01 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:01.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:01 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:43:01.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:01 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:02.248 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.261+0000 7f946a4cc700 1 -- 172.21.15.178:0/701298039 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9464102920 msgr2=0x7f94641091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:43:02.248 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.261+0000 7f946a4cc700 1 --2- 172.21.15.178:0/701298039 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9464102920 0x7f94641091d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f945c00b0a0 tx=0x7f945c00b3b0 comp rx=0 tx=0).stop 2024-02-21T23:43:02.248 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.261+0000 7f946a4cc700 1 -- 172.21.15.178:0/701298039 shutdown_connections 2024-02-21T23:43:02.248 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.261+0000 7f946a4cc700 1 --2- 172.21.15.178:0/701298039 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9464102920 0x7f94641091d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:43:02.248 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.261+0000 7f946a4cc700 1 --2- 172.21.15.178:0/701298039 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9464100d80 0x7f94641011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:43:02.248 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.261+0000 7f946a4cc700 1 --2- 172.21.15.178:0/701298039 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9464101f80 0x7f94641023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:43:02.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.261+0000 7f946a4cc700 1 -- 172.21.15.178:0/701298039 >> 172.21.15.178:0/701298039 conn(0x7f94640fc340 msgr2=0x7f94640fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:43:02.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.262+0000 7f946a4cc700 1 -- 172.21.15.178:0/701298039 shutdown_connections 2024-02-21T23:43:02.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.262+0000 7f946a4cc700 1 -- 172.21.15.178:0/701298039 wait complete. 2024-02-21T23:43:02.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.262+0000 7f946a4cc700 1 Processor -- start 2024-02-21T23:43:02.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.262+0000 7f946a4cc700 1 -- start start 2024-02-21T23:43:02.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.262+0000 7f946a4cc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9464100d80 0x7f9464196d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:43:02.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.262+0000 7f946a4cc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9464101f80 0x7f94641972c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:43:02.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.262+0000 7f946a4cc700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9464102920 0x7f946419e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:43:02.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.262+0000 7f946a4cc700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946419e880 con 0x7f9464101f80 2024-02-21T23:43:02.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.262+0000 7f946a4cc700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946419e9c0 con 0x7f9464100d80 2024-02-21T23:43:02.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.262+0000 7f946a4cc700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946419eb80 con 0x7f9464102920 2024-02-21T23:43:02.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.263+0000 7f9468a69700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9464102920 0x7f946419e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:43:02.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.263+0000 7f9468a69700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9464102920 0x7f946419e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:38000/0 (socket says 172.21.15.178:38000) 2024-02-21T23:43:02.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.263+0000 7f9468a69700 1 -- 172.21.15.178:0/337770738 learned_addr learned my addr 172.21.15.178:0/337770738 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-21T23:43:02.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.263+0000 7f94637fe700 1 --2- 172.21.15.178:0/337770738 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9464101f80 0x7f94641972c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:43:02.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.263+0000 7f9463fff700 1 --2- 172.21.15.178:0/337770738 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9464100d80 0x7f9464196d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:43:02.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.263+0000 7f9468a69700 1 -- 172.21.15.178:0/337770738 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9464100d80 msgr2=0x7f9464196d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:43:02.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.263+0000 7f9468a69700 1 --2- 172.21.15.178:0/337770738 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9464100d80 0x7f9464196d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:43:02.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.263+0000 7f9468a69700 1 -- 172.21.15.178:0/337770738 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9464101f80 msgr2=0x7f94641972c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:43:02.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.263+0000 7f9468a69700 1 --2- 172.21.15.178:0/337770738 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9464101f80 0x7f94641972c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:43:02.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.263+0000 7f9468a69700 1 -- 172.21.15.178:0/337770738 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f945c009d00 con 0x7f9464102920 2024-02-21T23:43:02.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.263+0000 7f94637fe700 1 --2- 172.21.15.178:0/337770738 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9464101f80 0x7f94641972c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:43:02.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.263+0000 7f9468a69700 1 --2- 172.21.15.178:0/337770738 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9464102920 0x7f946419e2b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f945c00b070 tx=0x7f945c00bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:43:02.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.263+0000 7f94617fa700 1 -- 172.21.15.178:0/337770738 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c00e040 con 0x7f9464102920 2024-02-21T23:43:02.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.263+0000 7f94617fa700 1 -- 172.21.15.178:0/337770738 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f945c004c50 con 0x7f9464102920 2024-02-21T23:43:02.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.263+0000 7f946a4cc700 1 -- 172.21.15.178:0/337770738 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9464073060 con 0x7f9464102920 2024-02-21T23:43:02.251 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.263+0000 7f94617fa700 1 -- 172.21.15.178:0/337770738 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c0052a0 con 0x7f9464102920 2024-02-21T23:43:02.251 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.263+0000 7f946a4cc700 1 -- 172.21.15.178:0/337770738 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94640735b0 con 0x7f9464102920 2024-02-21T23:43:02.251 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.264+0000 7f946a4cc700 1 -- 172.21.15.178:0/337770738 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9464108210 con 0x7f9464102920 2024-02-21T23:43:02.251 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.265+0000 7f94617fa700 1 -- 172.21.15.178:0/337770738 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 15) v1 ==== 135821+0+0 (secure 0 0 0) 0x7f945c0054c0 con 0x7f9464102920 2024-02-21T23:43:02.252 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.266+0000 7f94617fa700 1 --2- 172.21.15.178:0/337770738 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f94500a1110 0x7f94500a35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:43:02.252 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.266+0000 7f94617fa700 1 -- 172.21.15.178:0/337770738 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2029+0+0 (secure 0 0 0) 0x7f945c0cc470 con 0x7f9464102920 2024-02-21T23:43:02.252 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.266+0000 7f9463fff700 1 --2- 172.21.15.178:0/337770738 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f94500a1110 0x7f94500a35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:43:02.252 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.267+0000 7f9463fff700 1 --2- 172.21.15.178:0/337770738 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f94500a1110 0x7f94500a35d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9464198210 tx=0x7f944c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:43:02.254 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.271+0000 7f94617fa700 1 -- 172.21.15.178:0/337770738 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f945c097600 con 0x7f9464102920 2024-02-21T23:43:02.453 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:02.468+0000 7f946a4cc700 1 -- 172.21.15.178:0/337770738 --> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f94640611b0 con 0x7f94500a1110 2024-02-21T23:43:02.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:02 smithi178 ceph-mon[47547]: Detected new or changed devices on smithi176 2024-02-21T23:43:02.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:02 smithi178 ceph-mon[47547]: Adjusting osd_memory_target on smithi176 to 17114M 2024-02-21T23:43:02.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:02 smithi178 ceph-mon[47547]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-21T23:43:02.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:02 smithi178 ceph-mon[47547]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-02-21T23:43:02.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:02 smithi178 ceph-mon[47547]: osdmap e12: 2 total, 1 up, 2 in 2024-02-21T23:43:02.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:02 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-21T23:43:02.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:02 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-21T23:43:02.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:02 smithi090 ceph-mon[43385]: Detected new or changed devices on smithi176 2024-02-21T23:43:02.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:02 smithi090 ceph-mon[43385]: Adjusting osd_memory_target on smithi176 to 17114M 2024-02-21T23:43:02.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:02 smithi090 ceph-mon[43385]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-21T23:43:02.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:02 smithi090 ceph-mon[43385]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-02-21T23:43:02.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:02 smithi090 ceph-mon[43385]: osdmap e12: 2 total, 1 up, 2 in 2024-02-21T23:43:02.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:02 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-21T23:43:02.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:02 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-21T23:43:02.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:02 smithi176 ceph-mon[47172]: Detected new or changed devices on smithi176 2024-02-21T23:43:02.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:02 smithi176 ceph-mon[47172]: Adjusting osd_memory_target on smithi176 to 17114M 2024-02-21T23:43:02.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:02 smithi176 ceph-mon[47172]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-21T23:43:02.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:02 smithi176 ceph-mon[47172]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-02-21T23:43:02.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:02 smithi176 ceph-mon[47172]: osdmap e12: 2 total, 1 up, 2 in 2024-02-21T23:43:02.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:02 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-21T23:43:02.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:02 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-21T23:43:03.632 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:03 smithi178 ceph-mon[47547]: purged_snaps scrub starts 2024-02-21T23:43:03.632 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:03 smithi178 ceph-mon[47547]: purged_snaps scrub ok 2024-02-21T23:43:03.633 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:03 smithi178 ceph-mon[47547]: osd.1 [v2:172.21.15.176:6800/4085240925,v1:172.21.15.176:6801/4085240925] boot 2024-02-21T23:43:03.633 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:03 smithi178 ceph-mon[47547]: osdmap e13: 2 total, 2 up, 2 in 2024-02-21T23:43:03.633 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:03 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-21T23:43:03.633 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:03 smithi178 ceph-mon[47547]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:43:03.633 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:03 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-21T23:43:03.633 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:03 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-21T23:43:03.633 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:03 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:03.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:03 smithi090 ceph-mon[43385]: purged_snaps scrub starts 2024-02-21T23:43:03.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:03 smithi090 ceph-mon[43385]: purged_snaps scrub ok 2024-02-21T23:43:03.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:03 smithi090 ceph-mon[43385]: osd.1 [v2:172.21.15.176:6800/4085240925,v1:172.21.15.176:6801/4085240925] boot 2024-02-21T23:43:03.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:03 smithi090 ceph-mon[43385]: osdmap e13: 2 total, 2 up, 2 in 2024-02-21T23:43:03.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:03 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-21T23:43:03.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:03 smithi090 ceph-mon[43385]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:43:03.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:03 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-21T23:43:03.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:03 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-21T23:43:03.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:03 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:03.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:03 smithi176 ceph-mon[47172]: purged_snaps scrub starts 2024-02-21T23:43:03.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:03 smithi176 ceph-mon[47172]: purged_snaps scrub ok 2024-02-21T23:43:03.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:03 smithi176 ceph-mon[47172]: osd.1 [v2:172.21.15.176:6800/4085240925,v1:172.21.15.176:6801/4085240925] boot 2024-02-21T23:43:03.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:03 smithi176 ceph-mon[47172]: osdmap e13: 2 total, 2 up, 2 in 2024-02-21T23:43:03.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:03 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-21T23:43:03.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:03 smithi176 ceph-mon[47172]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:43:03.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:03 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-21T23:43:03.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:03 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-21T23:43:03.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:03 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:04.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:04 smithi178 ceph-mon[47547]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-21T23:43:04.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:04 smithi178 ceph-mon[47547]: osdmap e14: 2 total, 2 up, 2 in 2024-02-21T23:43:04.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:04 smithi090 ceph-mon[43385]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-21T23:43:04.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:04 smithi090 ceph-mon[43385]: osdmap e14: 2 total, 2 up, 2 in 2024-02-21T23:43:04.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:04 smithi176 ceph-mon[47172]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-21T23:43:04.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:04 smithi176 ceph-mon[47172]: osdmap e14: 2 total, 2 up, 2 in 2024-02-21T23:43:05.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:05 smithi178 ceph-mon[47547]: from='client.? 172.21.15.178:0/2874454957' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b83486bb-27f2-4d56-97fb-fb336fec4b99"}]: dispatch 2024-02-21T23:43:05.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:05 smithi178 ceph-mon[47547]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b83486bb-27f2-4d56-97fb-fb336fec4b99"}]: dispatch 2024-02-21T23:43:05.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:05 smithi178 ceph-mon[47547]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b83486bb-27f2-4d56-97fb-fb336fec4b99"}]': finished 2024-02-21T23:43:05.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:05 smithi178 ceph-mon[47547]: osdmap e15: 3 total, 2 up, 3 in 2024-02-21T23:43:05.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:05 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-21T23:43:05.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:05 smithi178 ceph-mon[47547]: from='client.? 172.21.15.178:0/3818710934' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-21T23:43:05.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:05 smithi090 ceph-mon[43385]: from='client.? 172.21.15.178:0/2874454957' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b83486bb-27f2-4d56-97fb-fb336fec4b99"}]: dispatch 2024-02-21T23:43:05.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:05 smithi090 ceph-mon[43385]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b83486bb-27f2-4d56-97fb-fb336fec4b99"}]: dispatch 2024-02-21T23:43:05.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:05 smithi090 ceph-mon[43385]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b83486bb-27f2-4d56-97fb-fb336fec4b99"}]': finished 2024-02-21T23:43:05.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:05 smithi090 ceph-mon[43385]: osdmap e15: 3 total, 2 up, 3 in 2024-02-21T23:43:05.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:05 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-21T23:43:05.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:05 smithi090 ceph-mon[43385]: from='client.? 172.21.15.178:0/3818710934' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-21T23:43:05.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:05 smithi176 ceph-mon[47172]: from='client.? 172.21.15.178:0/2874454957' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b83486bb-27f2-4d56-97fb-fb336fec4b99"}]: dispatch 2024-02-21T23:43:05.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:05 smithi176 ceph-mon[47172]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b83486bb-27f2-4d56-97fb-fb336fec4b99"}]: dispatch 2024-02-21T23:43:05.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:05 smithi176 ceph-mon[47172]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b83486bb-27f2-4d56-97fb-fb336fec4b99"}]': finished 2024-02-21T23:43:05.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:05 smithi176 ceph-mon[47172]: osdmap e15: 3 total, 2 up, 3 in 2024-02-21T23:43:05.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:05 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-21T23:43:05.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:05 smithi176 ceph-mon[47172]: from='client.? 172.21.15.178:0/3818710934' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-21T23:43:06.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:06 smithi178 ceph-mon[47547]: pgmap v51: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-21T23:43:06.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:06 smithi090 ceph-mon[43385]: pgmap v51: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-21T23:43:06.836 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:06 smithi176 ceph-mon[47172]: pgmap v51: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-21T23:43:07.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:07 smithi090 ceph-mon[43385]: pgmap v52: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-21T23:43:08.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:07 smithi176 ceph-mon[47172]: pgmap v52: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-21T23:43:08.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:07 smithi178 ceph-mon[47547]: pgmap v52: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-21T23:43:10.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:10 smithi178 ceph-mon[47547]: pgmap v53: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-21T23:43:10.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:10 smithi090 ceph-mon[43385]: pgmap v53: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-21T23:43:10.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:10 smithi176 ceph-mon[47172]: pgmap v53: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-21T23:43:11.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:11 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-21T23:43:11.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:11 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:11.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:11 smithi178 ceph-mon[47547]: Deploying daemon osd.2 on smithi178 2024-02-21T23:43:11.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:11 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-21T23:43:11.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:11 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:11.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:11 smithi090 ceph-mon[43385]: Deploying daemon osd.2 on smithi178 2024-02-21T23:43:11.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:11 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-21T23:43:11.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:11 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:11.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:11 smithi176 ceph-mon[47172]: Deploying daemon osd.2 on smithi178 2024-02-21T23:43:12.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:12 smithi178 ceph-mon[47547]: pgmap v54: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-21T23:43:12.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:12 smithi090 ceph-mon[43385]: pgmap v54: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-21T23:43:12.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:12 smithi176 ceph-mon[47172]: pgmap v54: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-21T23:43:14.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:14 smithi178 ceph-mon[47547]: pgmap v55: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-21T23:43:14.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:14 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:14.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:14 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:43:14.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:14 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:14.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:14 smithi090 ceph-mon[43385]: pgmap v55: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-21T23:43:14.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:14 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:14.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:14 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:43:14.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:14 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:14.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:14 smithi176 ceph-mon[47172]: pgmap v55: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-21T23:43:14.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:14 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:14.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:14 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:43:14.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:14 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:16.600 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:16 smithi178 ceph-mon[47547]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-21T23:43:16.600 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:16 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:16.600 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:16 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:16.600 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:16 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:16.600 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:16 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:43:16.600 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:16 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:16.660 INFO:teuthology.orchestra.run.smithi178.stdout:Created osd(s) 2 on host 'smithi178' 2024-02-21T23:43:16.662 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:16.670+0000 7f94617fa700 1 -- 172.21.15.178:0/337770738 <== mgr.14150 v2:172.21.15.90:6800/3697394685 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f94640611b0 con 0x7f94500a1110 2024-02-21T23:43:16.662 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:16.673+0000 7f946a4cc700 1 -- 172.21.15.178:0/337770738 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f94500a1110 msgr2=0x7f94500a35d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:43:16.662 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:16.673+0000 7f946a4cc700 1 --2- 172.21.15.178:0/337770738 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f94500a1110 0x7f94500a35d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9464198210 tx=0x7f944c009260 comp rx=0 tx=0).stop 2024-02-21T23:43:16.662 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:16.673+0000 7f946a4cc700 1 -- 172.21.15.178:0/337770738 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9464102920 msgr2=0x7f946419e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:43:16.662 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:16.673+0000 7f946a4cc700 1 --2- 172.21.15.178:0/337770738 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9464102920 0x7f946419e2b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f945c00b070 tx=0x7f945c00bab0 comp rx=0 tx=0).stop 2024-02-21T23:43:16.663 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:16.673+0000 7f946a4cc700 1 -- 172.21.15.178:0/337770738 shutdown_connections 2024-02-21T23:43:16.663 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:16.673+0000 7f946a4cc700 1 --2- 172.21.15.178:0/337770738 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f94500a1110 0x7f94500a35d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:43:16.663 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:16.673+0000 7f946a4cc700 1 --2- 172.21.15.178:0/337770738 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9464102920 0x7f946419e2b0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:43:16.663 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:16.673+0000 7f946a4cc700 1 --2- 172.21.15.178:0/337770738 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9464100d80 0x7f9464196d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:43:16.663 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:16.673+0000 7f946a4cc700 1 --2- 172.21.15.178:0/337770738 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9464101f80 0x7f94641972c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:43:16.663 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:16.673+0000 7f946a4cc700 1 -- 172.21.15.178:0/337770738 >> 172.21.15.178:0/337770738 conn(0x7f94640fc340 msgr2=0x7f9464105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:43:16.664 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:16.673+0000 7f946a4cc700 1 -- 172.21.15.178:0/337770738 shutdown_connections 2024-02-21T23:43:16.664 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:16.673+0000 7f946a4cc700 1 -- 172.21.15.178:0/337770738 wait complete. 2024-02-21T23:43:16.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:16 smithi090 ceph-mon[43385]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-21T23:43:16.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:16 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:16.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:16 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:16.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:16 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:16.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:16 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:43:16.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:16 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:16.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:16 smithi176 ceph-mon[47172]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-21T23:43:16.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:16 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:16.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:16 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:16.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:16 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:16.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:16 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:43:16.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:16 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:17.105 DEBUG:teuthology.orchestra.run.smithi178:osd.2> sudo journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@osd.2.service 2024-02-21T23:43:17.108 INFO:tasks.cephadm:Deploying osd.3 on smithi090 with /dev/vg_nvme/lv_3... 2024-02-21T23:43:17.109 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-21T23:43:17.140 INFO:journalctl@ceph.osd.2.smithi178.stdout:-- Logs begin at Wed 2024-02-21 23:24:26 UTC. -- 2024-02-21T23:43:17.342 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.a/config 2024-02-21T23:43:17.770 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:17 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:17.770 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:17 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:17.770 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:17 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:43:17.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:17 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:17.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:17 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:17.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:17 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:43:18.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:17 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:18.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:17 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:18.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:17 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:43:18.647 INFO:journalctl@ceph.osd.2.smithi178.stdout:Feb 21 23:43:18 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-2[53361]: 2024-02-21T23:43:18.362+0000 7facdf2e1640 -1 osd.2 0 log_to_monitors true 2024-02-21T23:43:18.667 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-21T23:43:18.667 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-21T23:43:18.667 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10+0 records in 2024-02-21T23:43:18.667 INFO:teuthology.orchestra.run.smithi090.stderr:10+0 records out 2024-02-21T23:43:18.667 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113304 s, 925 MB/s 2024-02-21T23:43:18.667 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping successful for: 2024-02-21T23:43:18.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:18 smithi178 ceph-mon[47547]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-21T23:43:18.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:18 smithi178 ceph-mon[47547]: from='osd.2 [v2:172.21.15.178:6800/3597250789,v1:172.21.15.178:6801/3597250789]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-21T23:43:18.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:18 smithi178 ceph-mon[47547]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-21T23:43:18.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:18 smithi090 ceph-mon[43385]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-21T23:43:18.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:18 smithi090 ceph-mon[43385]: from='osd.2 [v2:172.21.15.178:6800/3597250789,v1:172.21.15.178:6801/3597250789]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-21T23:43:18.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:18 smithi090 ceph-mon[43385]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-21T23:43:19.050 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph orch daemon add osd smithi090:vg_nvme/lv_3 2024-02-21T23:43:19.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:18 smithi176 ceph-mon[47172]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-21T23:43:19.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:18 smithi176 ceph-mon[47172]: from='osd.2 [v2:172.21.15.178:6800/3597250789,v1:172.21.15.178:6801/3597250789]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-21T23:43:19.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:18 smithi176 ceph-mon[47172]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-21T23:43:19.266 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.a/config 2024-02-21T23:43:19.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:19 smithi090 ceph-mon[43385]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-21T23:43:19.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:19 smithi090 ceph-mon[43385]: osdmap e16: 3 total, 2 up, 3 in 2024-02-21T23:43:19.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:19 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-21T23:43:19.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:19 smithi090 ceph-mon[43385]: from='osd.2 [v2:172.21.15.178:6800/3597250789,v1:172.21.15.178:6801/3597250789]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-02-21T23:43:19.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:19 smithi090 ceph-mon[43385]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-02-21T23:43:19.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:19 smithi090 ceph-mon[43385]: Detected new or changed devices on smithi178 2024-02-21T23:43:19.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:19 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:19.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:19 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:19.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:19 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:43:19.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:19 smithi090 ceph-mon[43385]: Adjusting osd_memory_target on smithi178 to 17113M 2024-02-21T23:43:19.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:19 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:19.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:19 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:19.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:19 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:43:19.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:19 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:19.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:19 smithi090 ceph-mon[43385]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-21T23:43:20.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:19 smithi176 ceph-mon[47172]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-21T23:43:20.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:19 smithi176 ceph-mon[47172]: osdmap e16: 3 total, 2 up, 3 in 2024-02-21T23:43:20.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:19 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-21T23:43:20.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:19 smithi176 ceph-mon[47172]: from='osd.2 [v2:172.21.15.178:6800/3597250789,v1:172.21.15.178:6801/3597250789]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-02-21T23:43:20.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:19 smithi176 ceph-mon[47172]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-02-21T23:43:20.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:19 smithi176 ceph-mon[47172]: Detected new or changed devices on smithi178 2024-02-21T23:43:20.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:19 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:20.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:19 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:20.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:19 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:43:20.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:19 smithi176 ceph-mon[47172]: Adjusting osd_memory_target on smithi178 to 17113M 2024-02-21T23:43:20.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:19 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:20.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:19 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:20.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:19 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:43:20.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:19 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:20.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:19 smithi176 ceph-mon[47172]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-21T23:43:20.174 INFO:journalctl@ceph.osd.2.smithi178.stdout:Feb 21 23:43:19 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-2[53361]: 2024-02-21T23:43:19.733+0000 7facd5146700 -1 osd.2 0 waiting for initial osdmap 2024-02-21T23:43:20.174 INFO:journalctl@ceph.osd.2.smithi178.stdout:Feb 21 23:43:19 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-2[53361]: 2024-02-21T23:43:19.737+0000 7faccdf35700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-21T23:43:20.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:19 smithi178 ceph-mon[47547]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-21T23:43:20.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:19 smithi178 ceph-mon[47547]: osdmap e16: 3 total, 2 up, 3 in 2024-02-21T23:43:20.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:19 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-21T23:43:20.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:19 smithi178 ceph-mon[47547]: from='osd.2 [v2:172.21.15.178:6800/3597250789,v1:172.21.15.178:6801/3597250789]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-02-21T23:43:20.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:19 smithi178 ceph-mon[47547]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-02-21T23:43:20.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:19 smithi178 ceph-mon[47547]: Detected new or changed devices on smithi178 2024-02-21T23:43:20.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:19 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:20.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:19 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:20.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:19 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:43:20.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:19 smithi178 ceph-mon[47547]: Adjusting osd_memory_target on smithi178 to 17113M 2024-02-21T23:43:20.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:19 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:20.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:19 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:20.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:19 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:43:20.176 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:19 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:20.176 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:19 smithi178 ceph-mon[47547]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-21T23:43:20.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.247+0000 7f3361ac6700 1 -- 172.21.15.90:0/3032400373 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f335c101830 msgr2=0x7f335c1080e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:43:20.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.247+0000 7f3361ac6700 1 --2- 172.21.15.90:0/3032400373 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f335c101830 0x7f335c1080e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3354009fc0 tx=0x7f335400b330 comp rx=0 tx=0).stop 2024-02-21T23:43:20.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.247+0000 7f3361ac6700 1 -- 172.21.15.90:0/3032400373 shutdown_connections 2024-02-21T23:43:20.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.247+0000 7f3361ac6700 1 --2- 172.21.15.90:0/3032400373 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f335c101830 0x7f335c1080e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:43:20.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.247+0000 7f3361ac6700 1 --2- 172.21.15.90:0/3032400373 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f335c0ffc90 0x7f335c1000b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:43:20.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.247+0000 7f3361ac6700 1 --2- 172.21.15.90:0/3032400373 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f335c100e90 0x7f335c1012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:43:20.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.247+0000 7f3361ac6700 1 -- 172.21.15.90:0/3032400373 >> 172.21.15.90:0/3032400373 conn(0x7f335c0fb230 msgr2=0x7f335c0fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:43:20.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.247+0000 7f3361ac6700 1 -- 172.21.15.90:0/3032400373 shutdown_connections 2024-02-21T23:43:20.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.247+0000 7f3361ac6700 1 -- 172.21.15.90:0/3032400373 wait complete. 2024-02-21T23:43:20.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.248+0000 7f3361ac6700 1 Processor -- start 2024-02-21T23:43:20.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.248+0000 7f3361ac6700 1 -- start start 2024-02-21T23:43:20.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.248+0000 7f3361ac6700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f335c0ffc90 0x7f335c0691f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:43:20.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.248+0000 7f3361ac6700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f335c100e90 0x7f335c19bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:43:20.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.248+0000 7f3361ac6700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f335c101830 0x7f335c19e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:43:20.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.248+0000 7f3361ac6700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f335c19e840 con 0x7f335c101830 2024-02-21T23:43:20.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.248+0000 7f3361ac6700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f335c19e9b0 con 0x7f335c0ffc90 2024-02-21T23:43:20.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.248+0000 7f3361ac6700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f335c19eb70 con 0x7f335c100e90 2024-02-21T23:43:20.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.248+0000 7f335bfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f335c101830 0x7f335c19e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:43:20.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.248+0000 7f335affd700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f335c100e90 0x7f335c19bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:43:20.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.248+0000 7f335bfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f335c101830 0x7f335c19e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51100/0 (socket says 172.21.15.90:51100) 2024-02-21T23:43:20.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.248+0000 7f335affd700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f335c100e90 0x7f335c19bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.90:34204/0 (socket says 172.21.15.90:34204) 2024-02-21T23:43:20.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.248+0000 7f335bfff700 1 -- 172.21.15.90:0/3956916116 learned_addr learned my addr 172.21.15.90:0/3956916116 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:43:20.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.248+0000 7f335b7fe700 1 --2- 172.21.15.90:0/3956916116 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f335c0ffc90 0x7f335c0691f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:43:20.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.248+0000 7f335bfff700 1 -- 172.21.15.90:0/3956916116 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f335c100e90 msgr2=0x7f335c19bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:43:20.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.248+0000 7f335bfff700 1 --2- 172.21.15.90:0/3956916116 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f335c100e90 0x7f335c19bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:43:20.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.248+0000 7f335bfff700 1 -- 172.21.15.90:0/3956916116 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f335c0ffc90 msgr2=0x7f335c0691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:43:20.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.248+0000 7f335bfff700 1 --2- 172.21.15.90:0/3956916116 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f335c0ffc90 0x7f335c0691f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:43:20.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.248+0000 7f335bfff700 1 -- 172.21.15.90:0/3956916116 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33440095f0 con 0x7f335c101830 2024-02-21T23:43:20.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.249+0000 7f335bfff700 1 --2- 172.21.15.90:0/3956916116 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f335c101830 0x7f335c19e0a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f33540077b0 tx=0x7f33540077e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:43:20.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.249+0000 7f3358ff9700 1 -- 172.21.15.90:0/3956916116 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f335400e040 con 0x7f335c101830 2024-02-21T23:43:20.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.249+0000 7f3361ac6700 1 -- 172.21.15.90:0/3956916116 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3354009c40 con 0x7f335c101830 2024-02-21T23:43:20.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.249+0000 7f3358ff9700 1 -- 172.21.15.90:0/3956916116 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3354004e70 con 0x7f335c101830 2024-02-21T23:43:20.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.249+0000 7f3358ff9700 1 -- 172.21.15.90:0/3956916116 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33540052a0 con 0x7f335c101830 2024-02-21T23:43:20.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.249+0000 7f3361ac6700 1 -- 172.21.15.90:0/3956916116 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f335c19f0d0 con 0x7f335c101830 2024-02-21T23:43:20.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.250+0000 7f3361ac6700 1 -- 172.21.15.90:0/3956916116 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f335c1070d0 con 0x7f335c101830 2024-02-21T23:43:20.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.251+0000 7f3358ff9700 1 -- 172.21.15.90:0/3956916116 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 15) v1 ==== 135821+0+0 (secure 0 0 0) 0x7f335401d470 con 0x7f335c101830 2024-02-21T23:43:20.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.252+0000 7f3358ff9700 1 --2- 172.21.15.90:0/3956916116 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f33480a1090 0x7f33480a3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:43:20.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.252+0000 7f335b7fe700 1 --2- 172.21.15.90:0/3956916116 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f33480a1090 0x7f33480a3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:43:20.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.252+0000 7f3358ff9700 1 -- 172.21.15.90:0/3956916116 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 2571+0+0 (secure 0 0 0) 0x7f33540cc530 con 0x7f335c101830 2024-02-21T23:43:20.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.252+0000 7f335b7fe700 1 --2- 172.21.15.90:0/3956916116 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f33480a1090 0x7f33480a3550 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f335c100cf0 tx=0x7f3344009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:43:20.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.256+0000 7f3358ff9700 1 -- 172.21.15.90:0/3956916116 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f33540974a0 con 0x7f335c101830 2024-02-21T23:43:20.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:20.458+0000 7f3361ac6700 1 -- 172.21.15.90:0/3956916116 --> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f335c0611b0 con 0x7f33480a1090 2024-02-21T23:43:20.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:20 smithi090 ceph-mon[43385]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-02-21T23:43:20.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:20 smithi090 ceph-mon[43385]: osdmap e17: 3 total, 2 up, 3 in 2024-02-21T23:43:20.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:20 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-21T23:43:20.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:20 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-21T23:43:20.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:20 smithi090 ceph-mon[43385]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:43:20.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:20 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-21T23:43:20.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:20 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-21T23:43:20.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:20 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:20.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:20 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-21T23:43:21.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:20 smithi176 ceph-mon[47172]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-02-21T23:43:21.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:20 smithi176 ceph-mon[47172]: osdmap e17: 3 total, 2 up, 3 in 2024-02-21T23:43:21.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:20 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-21T23:43:21.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:20 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-21T23:43:21.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:20 smithi176 ceph-mon[47172]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:43:21.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:20 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-21T23:43:21.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:20 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-21T23:43:21.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:20 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:21.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:20 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-21T23:43:21.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:20 smithi178 ceph-mon[47547]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-02-21T23:43:21.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:20 smithi178 ceph-mon[47547]: osdmap e17: 3 total, 2 up, 3 in 2024-02-21T23:43:21.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:20 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-21T23:43:21.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:20 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-21T23:43:21.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:20 smithi178 ceph-mon[47547]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:43:21.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:20 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-21T23:43:21.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:20 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-21T23:43:21.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:20 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:21.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:20 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-21T23:43:21.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:21 smithi090 ceph-mon[43385]: purged_snaps scrub starts 2024-02-21T23:43:21.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:21 smithi090 ceph-mon[43385]: purged_snaps scrub ok 2024-02-21T23:43:21.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:21 smithi090 ceph-mon[43385]: osd.2 [v2:172.21.15.178:6800/3597250789,v1:172.21.15.178:6801/3597250789] boot 2024-02-21T23:43:21.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:21 smithi090 ceph-mon[43385]: osdmap e18: 3 total, 3 up, 3 in 2024-02-21T23:43:21.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:21 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-21T23:43:21.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:21 smithi090 ceph-mon[43385]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-21T23:43:21.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:21 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-21T23:43:22.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:21 smithi176 ceph-mon[47172]: purged_snaps scrub starts 2024-02-21T23:43:22.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:21 smithi176 ceph-mon[47172]: purged_snaps scrub ok 2024-02-21T23:43:22.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:21 smithi176 ceph-mon[47172]: osd.2 [v2:172.21.15.178:6800/3597250789,v1:172.21.15.178:6801/3597250789] boot 2024-02-21T23:43:22.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:21 smithi176 ceph-mon[47172]: osdmap e18: 3 total, 3 up, 3 in 2024-02-21T23:43:22.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:21 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-21T23:43:22.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:21 smithi176 ceph-mon[47172]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-21T23:43:22.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:21 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-21T23:43:22.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:21 smithi178 ceph-mon[47547]: purged_snaps scrub starts 2024-02-21T23:43:22.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:21 smithi178 ceph-mon[47547]: purged_snaps scrub ok 2024-02-21T23:43:22.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:21 smithi178 ceph-mon[47547]: osd.2 [v2:172.21.15.178:6800/3597250789,v1:172.21.15.178:6801/3597250789] boot 2024-02-21T23:43:22.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:21 smithi178 ceph-mon[47547]: osdmap e18: 3 total, 3 up, 3 in 2024-02-21T23:43:22.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:21 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-21T23:43:22.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:21 smithi178 ceph-mon[47547]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-21T23:43:22.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:21 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-21T23:43:22.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:22 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-21T23:43:22.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:22 smithi090 ceph-mon[43385]: osdmap e19: 3 total, 3 up, 3 in 2024-02-21T23:43:22.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:22 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-21T23:43:22.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:22 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/3058285136' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "870e5bb6-eed5-4a0a-8e03-38de4f816b44"}]: dispatch 2024-02-21T23:43:22.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:22 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-21T23:43:22.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:22 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/3058285136' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "870e5bb6-eed5-4a0a-8e03-38de4f816b44"}]': finished 2024-02-21T23:43:22.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:22 smithi090 ceph-mon[43385]: osdmap e20: 4 total, 3 up, 4 in 2024-02-21T23:43:22.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:22 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-21T23:43:23.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:22 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-21T23:43:23.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:22 smithi176 ceph-mon[47172]: osdmap e19: 3 total, 3 up, 3 in 2024-02-21T23:43:23.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:22 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-21T23:43:23.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:22 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/3058285136' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "870e5bb6-eed5-4a0a-8e03-38de4f816b44"}]: dispatch 2024-02-21T23:43:23.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:22 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-21T23:43:23.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:22 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/3058285136' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "870e5bb6-eed5-4a0a-8e03-38de4f816b44"}]': finished 2024-02-21T23:43:23.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:22 smithi176 ceph-mon[47172]: osdmap e20: 4 total, 3 up, 4 in 2024-02-21T23:43:23.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:22 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-21T23:43:23.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:22 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-21T23:43:23.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:22 smithi178 ceph-mon[47547]: osdmap e19: 3 total, 3 up, 3 in 2024-02-21T23:43:23.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:22 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-21T23:43:23.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:22 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/3058285136' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "870e5bb6-eed5-4a0a-8e03-38de4f816b44"}]: dispatch 2024-02-21T23:43:23.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:22 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-21T23:43:23.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:22 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/3058285136' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "870e5bb6-eed5-4a0a-8e03-38de4f816b44"}]': finished 2024-02-21T23:43:23.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:22 smithi178 ceph-mon[47547]: osdmap e20: 4 total, 3 up, 4 in 2024-02-21T23:43:23.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:22 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-21T23:43:23.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:23 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/801592945' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-21T23:43:23.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:23 smithi090 ceph-mon[43385]: osdmap e21: 4 total, 3 up, 4 in 2024-02-21T23:43:23.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:23 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-21T23:43:23.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:23 smithi090 ceph-mon[43385]: pgmap v66: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-21T23:43:23.938 INFO:journalctl@ceph.osd.0.smithi090.stdout:Feb 21 23:43:23 smithi090 sudo[68741]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-21T23:43:23.939 INFO:journalctl@ceph.osd.0.smithi090.stdout:Feb 21 23:43:23 smithi090 sudo[68741]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-21T23:43:23.939 INFO:journalctl@ceph.osd.0.smithi090.stdout:Feb 21 23:43:23 smithi090 sudo[68741]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-02-21T23:43:23.939 INFO:journalctl@ceph.osd.0.smithi090.stdout:Feb 21 23:43:23 smithi090 sudo[68741]: pam_unix(sudo:session): session closed for user root 2024-02-21T23:43:24.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:23 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/801592945' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-21T23:43:24.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:23 smithi176 ceph-mon[47172]: osdmap e21: 4 total, 3 up, 4 in 2024-02-21T23:43:24.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:23 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-21T23:43:24.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:23 smithi176 ceph-mon[47172]: pgmap v66: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-21T23:43:24.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:23 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/801592945' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-21T23:43:24.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:23 smithi178 ceph-mon[47547]: osdmap e21: 4 total, 3 up, 4 in 2024-02-21T23:43:24.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:23 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-21T23:43:24.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:23 smithi178 ceph-mon[47547]: pgmap v66: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-21T23:43:24.227 INFO:journalctl@ceph.osd.0.smithi090.stdout:Feb 21 23:43:23 smithi090 sudo[68746]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-21T23:43:24.227 INFO:journalctl@ceph.osd.0.smithi090.stdout:Feb 21 23:43:23 smithi090 sudo[68746]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-21T23:43:24.227 INFO:journalctl@ceph.osd.0.smithi090.stdout:Feb 21 23:43:23 smithi090 sudo[68746]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-02-21T23:43:24.228 INFO:journalctl@ceph.osd.0.smithi090.stdout:Feb 21 23:43:23 smithi090 sudo[68746]: pam_unix(sudo:session): session closed for user root 2024-02-21T23:43:24.584 INFO:journalctl@ceph.osd.1.smithi176.stdout:Feb 21 23:43:24 smithi176 sudo[57698]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-21T23:43:24.584 INFO:journalctl@ceph.osd.1.smithi176.stdout:Feb 21 23:43:24 smithi176 sudo[57698]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-21T23:43:24.584 INFO:journalctl@ceph.osd.1.smithi176.stdout:Feb 21 23:43:24 smithi176 sudo[57698]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-02-21T23:43:24.584 INFO:journalctl@ceph.osd.1.smithi176.stdout:Feb 21 23:43:24 smithi176 sudo[57698]: pam_unix(sudo:session): session closed for user root 2024-02-21T23:43:24.584 INFO:journalctl@ceph.osd.1.smithi176.stdout:Feb 21 23:43:24 smithi176 sudo[57701]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-21T23:43:24.584 INFO:journalctl@ceph.osd.1.smithi176.stdout:Feb 21 23:43:24 smithi176 sudo[57701]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-21T23:43:24.584 INFO:journalctl@ceph.osd.1.smithi176.stdout:Feb 21 23:43:24 smithi176 sudo[57701]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-02-21T23:43:24.584 INFO:journalctl@ceph.osd.1.smithi176.stdout:Feb 21 23:43:24 smithi176 sudo[57701]: pam_unix(sudo:session): session closed for user root 2024-02-21T23:43:24.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:24.988+0000 7f3358ff9700 1 -- 172.21.15.90:0/3956916116 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7f3354097870 con 0x7f335c101830 2024-02-21T23:43:24.975 INFO:journalctl@ceph.osd.2.smithi178.stdout:Feb 21 23:43:24 smithi178 sudo[58798]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-21T23:43:24.975 INFO:journalctl@ceph.osd.2.smithi178.stdout:Feb 21 23:43:24 smithi178 sudo[58798]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-21T23:43:24.975 INFO:journalctl@ceph.osd.2.smithi178.stdout:Feb 21 23:43:24 smithi178 sudo[58798]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-02-21T23:43:24.975 INFO:journalctl@ceph.osd.2.smithi178.stdout:Feb 21 23:43:24 smithi178 sudo[58798]: pam_unix(sudo:session): session closed for user root 2024-02-21T23:43:24.975 INFO:journalctl@ceph.osd.2.smithi178.stdout:Feb 21 23:43:24 smithi178 sudo[58801]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-21T23:43:24.975 INFO:journalctl@ceph.osd.2.smithi178.stdout:Feb 21 23:43:24 smithi178 sudo[58801]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-21T23:43:24.975 INFO:journalctl@ceph.osd.2.smithi178.stdout:Feb 21 23:43:24 smithi178 sudo[58801]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-02-21T23:43:24.975 INFO:journalctl@ceph.osd.2.smithi178.stdout:Feb 21 23:43:24 smithi178 sudo[58801]: pam_unix(sudo:session): session closed for user root 2024-02-21T23:43:25.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:24 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:25.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:24 smithi090 ceph-mon[43385]: osdmap e22: 4 total, 3 up, 4 in 2024-02-21T23:43:25.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:24 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-21T23:43:25.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:24 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:25.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:24 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:25.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:24 smithi176 ceph-mon[47172]: osdmap e22: 4 total, 3 up, 4 in 2024-02-21T23:43:25.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:24 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-21T23:43:25.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:24 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:25.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:24 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:25.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:24 smithi178 ceph-mon[47547]: osdmap e22: 4 total, 3 up, 4 in 2024-02-21T23:43:25.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:24 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-21T23:43:25.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:24 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:25.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:25 smithi090 sudo[69497]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-21T23:43:25.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:25 smithi090 sudo[69497]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-21T23:43:25.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:25 smithi090 sudo[69497]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-02-21T23:43:25.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:25 smithi090 sudo[69497]: pam_unix(sudo:session): session closed for user root 2024-02-21T23:43:25.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:25 smithi090 sudo[69502]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-21T23:43:25.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:25 smithi090 sudo[69502]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-21T23:43:25.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:25 smithi090 sudo[69502]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-02-21T23:43:25.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:25 smithi090 sudo[69502]: pam_unix(sudo:session): session closed for user root 2024-02-21T23:43:26.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:25 smithi178 sudo[58804]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-21T23:43:26.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:25 smithi178 sudo[58804]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-21T23:43:26.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:25 smithi178 sudo[58804]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-02-21T23:43:26.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:25 smithi178 sudo[58804]: pam_unix(sudo:session): session closed for user root 2024-02-21T23:43:26.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:25 smithi178 sudo[58807]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-21T23:43:26.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:25 smithi178 sudo[58807]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-21T23:43:26.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:25 smithi178 sudo[58807]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-02-21T23:43:26.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:25 smithi178 sudo[58807]: pam_unix(sudo:session): session closed for user root 2024-02-21T23:43:26.443 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:26 smithi176 sudo[57704]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-21T23:43:26.443 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:26 smithi176 sudo[57704]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-21T23:43:26.443 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:26 smithi176 sudo[57704]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-02-21T23:43:26.443 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:26 smithi176 sudo[57704]: pam_unix(sudo:session): session closed for user root 2024-02-21T23:43:26.443 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:26 smithi176 sudo[57707]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-21T23:43:26.443 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:26 smithi176 sudo[57707]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-21T23:43:26.443 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:26 smithi176 sudo[57707]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-02-21T23:43:26.443 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:26 smithi176 sudo[57707]: pam_unix(sudo:session): session closed for user root 2024-02-21T23:43:26.444 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:26 smithi176 ceph-mon[47172]: mgrmap e16: x(active, since 111s), standbys: z, y 2024-02-21T23:43:26.444 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:26 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:26.444 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:26 smithi176 ceph-mon[47172]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-21T23:43:26.444 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:26 smithi176 ceph-mon[47172]: pgmap v68: 1 pgs: 1 active+clean; 257 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-21T23:43:26.444 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:26 smithi176 ceph-mon[47172]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-21T23:43:26.444 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:26 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-21T23:43:26.444 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:26 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-21T23:43:26.444 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:26 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-21T23:43:26.444 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:26 smithi176 ceph-mon[47172]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-21T23:43:26.444 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:26 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-21T23:43:26.444 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:26 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-21T23:43:26.444 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:26 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-21T23:43:26.444 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:26 smithi176 ceph-mon[47172]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-21T23:43:26.445 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:26 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-21T23:43:26.445 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:26 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-21T23:43:26.445 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:26 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-21T23:43:26.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:26 smithi090 ceph-mon[43385]: mgrmap e16: x(active, since 111s), standbys: z, y 2024-02-21T23:43:26.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:26 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:26.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:26 smithi090 ceph-mon[43385]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-21T23:43:26.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:26 smithi090 ceph-mon[43385]: pgmap v68: 1 pgs: 1 active+clean; 257 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-21T23:43:26.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:26 smithi090 ceph-mon[43385]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-21T23:43:26.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:26 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-21T23:43:26.729 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:26 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-21T23:43:26.729 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:26 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-21T23:43:26.729 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:26 smithi090 ceph-mon[43385]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-21T23:43:26.729 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:26 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-21T23:43:26.729 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:26 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-21T23:43:26.730 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:26 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-21T23:43:26.730 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:26 smithi090 ceph-mon[43385]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-21T23:43:26.730 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:26 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-21T23:43:26.730 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:26 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-21T23:43:26.730 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:26 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-21T23:43:26.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:26 smithi178 ceph-mon[47547]: mgrmap e16: x(active, since 111s), standbys: z, y 2024-02-21T23:43:26.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:26 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:26.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:26 smithi178 ceph-mon[47547]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-21T23:43:26.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:26 smithi178 ceph-mon[47547]: pgmap v68: 1 pgs: 1 active+clean; 257 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-21T23:43:26.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:26 smithi178 ceph-mon[47547]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-21T23:43:26.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:26 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-21T23:43:26.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:26 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-21T23:43:26.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:26 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-21T23:43:26.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:26 smithi178 ceph-mon[47547]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-21T23:43:26.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:26 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-21T23:43:26.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:26 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-21T23:43:26.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:26 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-21T23:43:26.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:26 smithi178 ceph-mon[47547]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-21T23:43:26.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:26 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-21T23:43:26.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:26 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-21T23:43:26.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:26 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-21T23:43:27.706 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:27 smithi090 ceph-mon[43385]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-21T23:43:27.706 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:27 smithi090 ceph-mon[43385]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-21T23:43:27.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:27 smithi176 ceph-mon[47172]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-21T23:43:27.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:27 smithi176 ceph-mon[47172]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-21T23:43:27.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:27 smithi178 ceph-mon[47547]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-21T23:43:27.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:27 smithi178 ceph-mon[47547]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-21T23:43:28.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:28 smithi090 ceph-mon[43385]: pgmap v69: 1 pgs: 1 active+clean; 257 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-21T23:43:28.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:28 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-21T23:43:28.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:28 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:28.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:28 smithi176 ceph-mon[47172]: pgmap v69: 1 pgs: 1 active+clean; 257 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-21T23:43:28.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:28 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-21T23:43:28.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:28 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:28.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:28 smithi178 ceph-mon[47547]: pgmap v69: 1 pgs: 1 active+clean; 257 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-21T23:43:28.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:28 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-21T23:43:28.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:28 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:29.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:29 smithi090 ceph-mon[43385]: Deploying daemon osd.3 on smithi090 2024-02-21T23:43:29.833 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:29 smithi176 ceph-mon[47172]: Deploying daemon osd.3 on smithi090 2024-02-21T23:43:29.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:29 smithi178 ceph-mon[47547]: Deploying daemon osd.3 on smithi090 2024-02-21T23:43:30.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:30 smithi090 ceph-mon[43385]: pgmap v70: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 16 KiB/s rd, 485 KiB/s wr, 41 op/s 2024-02-21T23:43:30.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:30 smithi176 ceph-mon[47172]: pgmap v70: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 16 KiB/s rd, 485 KiB/s wr, 41 op/s 2024-02-21T23:43:30.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:30 smithi178 ceph-mon[47547]: pgmap v70: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 16 KiB/s rd, 485 KiB/s wr, 41 op/s 2024-02-21T23:43:32.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:32 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:32.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:32 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:32.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:32 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:43:32.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:32 smithi090 ceph-mon[43385]: pgmap v71: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 11 KiB/s rd, 351 KiB/s wr, 30 op/s 2024-02-21T23:43:32.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:32 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:32.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:32 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:32.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:32 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:43:32.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:32 smithi176 ceph-mon[47172]: pgmap v71: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 11 KiB/s rd, 351 KiB/s wr, 30 op/s 2024-02-21T23:43:32.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:32 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:32.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:32 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:32.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:32 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:43:32.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:32 smithi178 ceph-mon[47547]: pgmap v71: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 11 KiB/s rd, 351 KiB/s wr, 30 op/s 2024-02-21T23:43:33.534 INFO:teuthology.orchestra.run.smithi090.stdout:Created osd(s) 3 on host 'smithi090' 2024-02-21T23:43:33.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:33.549+0000 7f3358ff9700 1 -- 172.21.15.90:0/3956916116 <== mgr.14150 v2:172.21.15.90:6800/3697394685 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f335c0611b0 con 0x7f33480a1090 2024-02-21T23:43:33.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:33.551+0000 7f3361ac6700 1 -- 172.21.15.90:0/3956916116 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f33480a1090 msgr2=0x7f33480a3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:43:33.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:33.551+0000 7f3361ac6700 1 --2- 172.21.15.90:0/3956916116 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f33480a1090 0x7f33480a3550 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f335c100cf0 tx=0x7f3344009310 comp rx=0 tx=0).stop 2024-02-21T23:43:33.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:33.551+0000 7f3361ac6700 1 -- 172.21.15.90:0/3956916116 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f335c101830 msgr2=0x7f335c19e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:43:33.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:33.551+0000 7f3361ac6700 1 --2- 172.21.15.90:0/3956916116 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f335c101830 0x7f335c19e0a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f33540077b0 tx=0x7f33540077e0 comp rx=0 tx=0).stop 2024-02-21T23:43:33.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:33.551+0000 7f3361ac6700 1 -- 172.21.15.90:0/3956916116 shutdown_connections 2024-02-21T23:43:33.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:33.551+0000 7f3361ac6700 1 --2- 172.21.15.90:0/3956916116 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f33480a1090 0x7f33480a3550 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:43:33.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:33.551+0000 7f3361ac6700 1 --2- 172.21.15.90:0/3956916116 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f335c101830 0x7f335c19e0a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:43:33.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:33.551+0000 7f3361ac6700 1 --2- 172.21.15.90:0/3956916116 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f335c0ffc90 0x7f335c0691f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:43:33.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:33.551+0000 7f3361ac6700 1 --2- 172.21.15.90:0/3956916116 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f335c100e90 0x7f335c19bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:43:33.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:33.551+0000 7f3361ac6700 1 -- 172.21.15.90:0/3956916116 >> 172.21.15.90:0/3956916116 conn(0x7f335c0fb230 msgr2=0x7f335c104210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:43:33.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:33.552+0000 7f3361ac6700 1 -- 172.21.15.90:0/3956916116 shutdown_connections 2024-02-21T23:43:33.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:43:33.552+0000 7f3361ac6700 1 -- 172.21.15.90:0/3956916116 wait complete. 2024-02-21T23:43:33.703 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:33 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:33.703 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:33 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:33.703 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:33 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:33.703 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:33 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:43:33.704 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:33 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:33.704 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:33 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:33.704 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:33 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:33.704 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:33 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:43:33.921 DEBUG:teuthology.orchestra.run.smithi090:osd.3> sudo journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@osd.3.service 2024-02-21T23:43:33.923 INFO:tasks.cephadm:Deploying osd.4 on smithi176 with /dev/vg_nvme/lv_3... 2024-02-21T23:43:33.923 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-21T23:43:33.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:33 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:33.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:33 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:33.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:33 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:33.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:33 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:43:33.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:33 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:33.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:33 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:33.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:33 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:33.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:33 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:43:33.955 INFO:journalctl@ceph.osd.3.smithi090.stdout:-- Logs begin at Wed 2024-02-21 23:24:49 UTC. -- 2024-02-21T23:43:33.957 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:33 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:33.957 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:33 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:33.957 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:33 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:33.957 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:33 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:43:33.958 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:33 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:33.958 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:33 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:33.958 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:33 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:33.958 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:33 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:43:34.157 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.b/config 2024-02-21T23:43:34.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:34 smithi090 ceph-mon[43385]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 8.9 KiB/s rd, 275 KiB/s wr, 23 op/s 2024-02-21T23:43:34.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:34 smithi176 ceph-mon[47172]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 8.9 KiB/s rd, 275 KiB/s wr, 23 op/s 2024-02-21T23:43:34.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:34 smithi178 ceph-mon[47547]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 8.9 KiB/s rd, 275 KiB/s wr, 23 op/s 2024-02-21T23:43:35.399 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-21T23:43:35.399 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-21T23:43:35.399 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10+0 records in 2024-02-21T23:43:35.399 INFO:teuthology.orchestra.run.smithi176.stderr:10+0 records out 2024-02-21T23:43:35.399 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112718 s, 930 MB/s 2024-02-21T23:43:35.399 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping successful for: 2024-02-21T23:43:35.477 INFO:journalctl@ceph.osd.3.smithi090.stdout:Feb 21 23:43:35 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-3[70542]: 2024-02-21T23:43:35.174+0000 7f2577429640 -1 osd.3 0 log_to_monitors true 2024-02-21T23:43:35.716 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:35 smithi176 ceph-mon[47172]: from='osd.3 [v2:172.21.15.90:6810/3299043821,v1:172.21.15.90:6811/3299043821]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-21T23:43:35.775 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph orch daemon add osd smithi176:vg_nvme/lv_3 2024-02-21T23:43:35.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:35 smithi178 ceph-mon[47547]: from='osd.3 [v2:172.21.15.90:6810/3299043821,v1:172.21.15.90:6811/3299043821]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-21T23:43:35.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:35 smithi090 ceph-mon[43385]: from='osd.3 [v2:172.21.15.90:6810/3299043821,v1:172.21.15.90:6811/3299043821]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-21T23:43:36.033 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.b/config 2024-02-21T23:43:36.797 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:36 smithi176 ceph-mon[47172]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 7.3 KiB/s rd, 226 KiB/s wr, 19 op/s 2024-02-21T23:43:36.797 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:36 smithi176 ceph-mon[47172]: from='osd.3 [v2:172.21.15.90:6810/3299043821,v1:172.21.15.90:6811/3299043821]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-21T23:43:36.798 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:36 smithi176 ceph-mon[47172]: osdmap e23: 4 total, 3 up, 4 in 2024-02-21T23:43:36.798 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:36 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-21T23:43:36.798 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:36 smithi176 ceph-mon[47172]: from='osd.3 [v2:172.21.15.90:6810/3299043821,v1:172.21.15.90:6811/3299043821]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-21T23:43:36.798 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:36 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:36.798 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:36 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:36.798 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:36 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:43:36.798 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:36 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:43:36.798 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:36 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:36.798 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:36 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:36.798 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:36 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:43:36.798 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:36 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:36.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:36 smithi178 ceph-mon[47547]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 7.3 KiB/s rd, 226 KiB/s wr, 19 op/s 2024-02-21T23:43:36.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:36 smithi178 ceph-mon[47547]: from='osd.3 [v2:172.21.15.90:6810/3299043821,v1:172.21.15.90:6811/3299043821]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-21T23:43:36.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:36 smithi178 ceph-mon[47547]: osdmap e23: 4 total, 3 up, 4 in 2024-02-21T23:43:36.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:36 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-21T23:43:36.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:36 smithi178 ceph-mon[47547]: from='osd.3 [v2:172.21.15.90:6810/3299043821,v1:172.21.15.90:6811/3299043821]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-21T23:43:36.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:36 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:36.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:36 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:36.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:36 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:43:36.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:36 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:43:36.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:36 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:36.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:36 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:36.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:36 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:43:36.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:36 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:36.976 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.990+0000 7fcfa4da2700 1 -- 172.21.15.176:0/2669422571 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfa0100e90 msgr2=0x7fcfa01012f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:43:36.976 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.990+0000 7fcfa4da2700 1 --2- 172.21.15.176:0/2669422571 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfa0100e90 0x7fcfa01012f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fcf900099c0 tx=0x7fcf90009cd0 comp rx=0 tx=0).stop 2024-02-21T23:43:36.976 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.990+0000 7fcfa4da2700 1 -- 172.21.15.176:0/2669422571 shutdown_connections 2024-02-21T23:43:36.976 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.990+0000 7fcfa4da2700 1 --2- 172.21.15.176:0/2669422571 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcfa0101830 0x7fcfa01080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:43:36.976 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.990+0000 7fcfa4da2700 1 --2- 172.21.15.176:0/2669422571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfa00ffc90 0x7fcfa01000b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:43:36.976 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.990+0000 7fcfa4da2700 1 --2- 172.21.15.176:0/2669422571 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfa0100e90 0x7fcfa01012f0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:43:36.977 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.990+0000 7fcfa4da2700 1 -- 172.21.15.176:0/2669422571 >> 172.21.15.176:0/2669422571 conn(0x7fcfa00fb230 msgr2=0x7fcfa00fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:43:36.977 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.990+0000 7fcfa4da2700 1 -- 172.21.15.176:0/2669422571 shutdown_connections 2024-02-21T23:43:36.977 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.990+0000 7fcfa4da2700 1 -- 172.21.15.176:0/2669422571 wait complete. 2024-02-21T23:43:36.977 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.991+0000 7fcfa4da2700 1 Processor -- start 2024-02-21T23:43:36.977 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.991+0000 7fcfa4da2700 1 -- start start 2024-02-21T23:43:36.977 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.991+0000 7fcfa4da2700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcfa00ffc90 0x7fcfa0196d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:43:36.977 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.991+0000 7fcfa4da2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfa0100e90 0x7fcfa0197240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:43:36.977 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.991+0000 7fcfa4da2700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfa0101830 0x7fcfa019e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:43:36.977 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.991+0000 7fcfa4da2700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa019e7e0 con 0x7fcfa0100e90 2024-02-21T23:43:36.977 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.991+0000 7fcfa4da2700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa019e920 con 0x7fcfa0101830 2024-02-21T23:43:36.977 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.991+0000 7fcfa4da2700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa019eae0 con 0x7fcfa00ffc90 2024-02-21T23:43:36.977 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.991+0000 7fcf9ed9d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfa0101830 0x7fcfa019e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:43:36.977 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.991+0000 7fcf9ed9d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfa0101830 0x7fcfa019e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:58518/0 (socket says 172.21.15.176:58518) 2024-02-21T23:43:36.978 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.991+0000 7fcf9ed9d700 1 -- 172.21.15.176:0/2746837197 learned_addr learned my addr 172.21.15.176:0/2746837197 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-21T23:43:36.978 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.991+0000 7fcf9e59c700 1 --2- 172.21.15.176:0/2746837197 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcfa00ffc90 0x7fcfa0196d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:43:36.978 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.991+0000 7fcf9dd9b700 1 --2- 172.21.15.176:0/2746837197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfa0100e90 0x7fcfa0197240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:43:36.978 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.991+0000 7fcf9ed9d700 1 -- 172.21.15.176:0/2746837197 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcfa00ffc90 msgr2=0x7fcfa0196d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:43:36.978 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.991+0000 7fcf9ed9d700 1 --2- 172.21.15.176:0/2746837197 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcfa00ffc90 0x7fcfa0196d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:43:36.978 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.991+0000 7fcf9ed9d700 1 -- 172.21.15.176:0/2746837197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfa0100e90 msgr2=0x7fcfa0197240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:43:36.978 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.991+0000 7fcf9ed9d700 1 --2- 172.21.15.176:0/2746837197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfa0100e90 0x7fcfa0197240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:43:36.978 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.991+0000 7fcf9ed9d700 1 -- 172.21.15.176:0/2746837197 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf900095f0 con 0x7fcfa0101830 2024-02-21T23:43:36.978 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.991+0000 7fcf9dd9b700 1 --2- 172.21.15.176:0/2746837197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfa0100e90 0x7fcfa0197240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:43:36.978 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.992+0000 7fcf9ed9d700 1 --2- 172.21.15.176:0/2746837197 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfa0101830 0x7fcfa019e210 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fcf9800ada0 tx=0x7fcf9800c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:43:36.978 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.992+0000 7fcf8f7fe700 1 -- 172.21.15.176:0/2746837197 <== mon.2 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9800f070 con 0x7fcfa0101830 2024-02-21T23:43:36.978 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.992+0000 7fcfa4da2700 1 -- 172.21.15.176:0/2746837197 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfa019edc0 con 0x7fcfa0101830 2024-02-21T23:43:36.978 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.992+0000 7fcf8f7fe700 1 -- 172.21.15.176:0/2746837197 <== mon.2 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcf98007b30 con 0x7fcfa0101830 2024-02-21T23:43:36.978 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.992+0000 7fcf8f7fe700 1 -- 172.21.15.176:0/2746837197 <== mon.2 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf98005440 con 0x7fcfa0101830 2024-02-21T23:43:36.979 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.992+0000 7fcfa4da2700 1 -- 172.21.15.176:0/2746837197 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfa019f240 con 0x7fcfa0101830 2024-02-21T23:43:36.979 INFO:journalctl@ceph.osd.3.smithi090.stdout:Feb 21 23:43:36 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-3[70542]: 2024-02-21T23:43:36.644+0000 7f256ca8d700 -1 osd.3 0 waiting for initial osdmap 2024-02-21T23:43:36.979 INFO:journalctl@ceph.osd.3.smithi090.stdout:Feb 21 23:43:36 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-3[70542]: 2024-02-21T23:43:36.650+0000 7f2568882700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-21T23:43:36.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:36 smithi090 ceph-mon[43385]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 7.3 KiB/s rd, 226 KiB/s wr, 19 op/s 2024-02-21T23:43:36.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:36 smithi090 ceph-mon[43385]: from='osd.3 [v2:172.21.15.90:6810/3299043821,v1:172.21.15.90:6811/3299043821]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-21T23:43:36.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:36 smithi090 ceph-mon[43385]: osdmap e23: 4 total, 3 up, 4 in 2024-02-21T23:43:36.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:36 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-21T23:43:36.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:36 smithi090 ceph-mon[43385]: from='osd.3 [v2:172.21.15.90:6810/3299043821,v1:172.21.15.90:6811/3299043821]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-21T23:43:36.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:36 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:36.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:36 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:36.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:36 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:43:36.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:36 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:43:36.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:36 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:36.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:36 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:36.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:36 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:43:36.981 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:36 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:36.981 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.993+0000 7fcfa4da2700 1 -- 172.21.15.176:0/2746837197 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfa004ea40 con 0x7fcfa0101830 2024-02-21T23:43:36.981 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.994+0000 7fcf8f7fe700 1 -- 172.21.15.176:0/2746837197 <== mon.2 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7fcf98004ad0 con 0x7fcfa0101830 2024-02-21T23:43:36.982 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.994+0000 7fcf8f7fe700 1 --2- 172.21.15.176:0/2746837197 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fcf880a1120 0x7fcf880a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:43:36.982 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.994+0000 7fcf8f7fe700 1 -- 172.21.15.176:0/2746837197 <== mon.2 v2:172.21.15.176:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 3702+0+0 (secure 0 0 0) 0x7fcf980cb330 con 0x7fcfa0101830 2024-02-21T23:43:36.982 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.995+0000 7fcf9e59c700 1 --2- 172.21.15.176:0/2746837197 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fcf880a1120 0x7fcf880a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:43:36.982 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.995+0000 7fcf9e59c700 1 --2- 172.21.15.176:0/2746837197 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fcf880a1120 0x7fcf880a35e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcf94005950 tx=0x7fcf940058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:43:36.982 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:36.998+0000 7fcf8f7fe700 1 -- 172.21.15.176:0/2746837197 <== mon.2 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fcf98095d50 con 0x7fcfa0101830 2024-02-21T23:43:37.181 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:37.196+0000 7fcfa4da2700 1 -- 172.21.15.176:0/2746837197 --> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fcfa0068df0 con 0x7fcf880a1120 2024-02-21T23:43:37.804 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:37 smithi176 ceph-mon[47172]: Detected new or changed devices on smithi090 2024-02-21T23:43:37.804 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:37 smithi176 ceph-mon[47172]: Adjusting osd_memory_target on smithi090 to 8556M 2024-02-21T23:43:37.804 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:37 smithi176 ceph-mon[47172]: from='osd.3 [v2:172.21.15.90:6810/3299043821,v1:172.21.15.90:6811/3299043821]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-02-21T23:43:37.804 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:37 smithi176 ceph-mon[47172]: osdmap e24: 4 total, 3 up, 4 in 2024-02-21T23:43:37.804 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:37 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-21T23:43:37.804 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:37 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-21T23:43:37.804 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:37 smithi176 ceph-mon[47172]: osd.3 [v2:172.21.15.90:6810/3299043821,v1:172.21.15.90:6811/3299043821] boot 2024-02-21T23:43:37.804 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:37 smithi176 ceph-mon[47172]: osdmap e25: 4 total, 4 up, 4 in 2024-02-21T23:43:37.804 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:37 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-21T23:43:37.804 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:37 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-21T23:43:37.805 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:37 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-21T23:43:37.805 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:37 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:37.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:37 smithi178 ceph-mon[47547]: Detected new or changed devices on smithi090 2024-02-21T23:43:37.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:37 smithi178 ceph-mon[47547]: Adjusting osd_memory_target on smithi090 to 8556M 2024-02-21T23:43:37.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:37 smithi178 ceph-mon[47547]: from='osd.3 [v2:172.21.15.90:6810/3299043821,v1:172.21.15.90:6811/3299043821]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-02-21T23:43:37.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:37 smithi178 ceph-mon[47547]: osdmap e24: 4 total, 3 up, 4 in 2024-02-21T23:43:37.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:37 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-21T23:43:37.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:37 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-21T23:43:37.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:37 smithi178 ceph-mon[47547]: osd.3 [v2:172.21.15.90:6810/3299043821,v1:172.21.15.90:6811/3299043821] boot 2024-02-21T23:43:37.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:37 smithi178 ceph-mon[47547]: osdmap e25: 4 total, 4 up, 4 in 2024-02-21T23:43:37.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:37 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-21T23:43:37.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:37 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-21T23:43:37.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:37 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-21T23:43:37.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:37 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:37.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:37 smithi090 ceph-mon[43385]: Detected new or changed devices on smithi090 2024-02-21T23:43:37.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:37 smithi090 ceph-mon[43385]: Adjusting osd_memory_target on smithi090 to 8556M 2024-02-21T23:43:37.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:37 smithi090 ceph-mon[43385]: from='osd.3 [v2:172.21.15.90:6810/3299043821,v1:172.21.15.90:6811/3299043821]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-02-21T23:43:37.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:37 smithi090 ceph-mon[43385]: osdmap e24: 4 total, 3 up, 4 in 2024-02-21T23:43:37.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:37 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-21T23:43:37.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:37 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-21T23:43:37.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:37 smithi090 ceph-mon[43385]: osd.3 [v2:172.21.15.90:6810/3299043821,v1:172.21.15.90:6811/3299043821] boot 2024-02-21T23:43:37.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:37 smithi090 ceph-mon[43385]: osdmap e25: 4 total, 4 up, 4 in 2024-02-21T23:43:37.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:37 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-21T23:43:37.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:37 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-21T23:43:37.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:37 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-21T23:43:37.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:37 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:38.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:38 smithi178 ceph-mon[47547]: purged_snaps scrub starts 2024-02-21T23:43:38.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:38 smithi178 ceph-mon[47547]: purged_snaps scrub ok 2024-02-21T23:43:38.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:38 smithi178 ceph-mon[47547]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:43:38.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:38 smithi178 ceph-mon[47547]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-21T23:43:38.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:38 smithi178 ceph-mon[47547]: osdmap e26: 4 total, 4 up, 4 in 2024-02-21T23:43:38.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:38 smithi090 ceph-mon[43385]: purged_snaps scrub starts 2024-02-21T23:43:38.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:38 smithi090 ceph-mon[43385]: purged_snaps scrub ok 2024-02-21T23:43:38.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:38 smithi090 ceph-mon[43385]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:43:38.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:38 smithi090 ceph-mon[43385]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-21T23:43:38.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:38 smithi090 ceph-mon[43385]: osdmap e26: 4 total, 4 up, 4 in 2024-02-21T23:43:39.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:38 smithi176 ceph-mon[47172]: purged_snaps scrub starts 2024-02-21T23:43:39.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:38 smithi176 ceph-mon[47172]: purged_snaps scrub ok 2024-02-21T23:43:39.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:38 smithi176 ceph-mon[47172]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:43:39.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:38 smithi176 ceph-mon[47172]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-21T23:43:39.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:38 smithi176 ceph-mon[47172]: osdmap e26: 4 total, 4 up, 4 in 2024-02-21T23:43:39.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:39 smithi090 ceph-mon[43385]: osdmap e27: 4 total, 4 up, 4 in 2024-02-21T23:43:39.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:39 smithi090 ceph-mon[43385]: from='client.? 172.21.15.176:0/1632227461' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cb9f7f48-76e0-4368-ab87-2a09a8487ccd"}]: dispatch 2024-02-21T23:43:39.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:39 smithi090 ceph-mon[43385]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cb9f7f48-76e0-4368-ab87-2a09a8487ccd"}]: dispatch 2024-02-21T23:43:39.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:39 smithi090 ceph-mon[43385]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cb9f7f48-76e0-4368-ab87-2a09a8487ccd"}]': finished 2024-02-21T23:43:39.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:39 smithi090 ceph-mon[43385]: osdmap e28: 5 total, 4 up, 5 in 2024-02-21T23:43:39.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:39 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-21T23:43:39.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:39 smithi090 ceph-mon[43385]: pgmap v81: 1 pgs: 1 remapped+peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-02-21T23:43:39.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:39 smithi090 ceph-mon[43385]: from='client.? 172.21.15.176:0/454335876' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-21T23:43:40.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:39 smithi176 ceph-mon[47172]: osdmap e27: 4 total, 4 up, 4 in 2024-02-21T23:43:40.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:39 smithi176 ceph-mon[47172]: from='client.? 172.21.15.176:0/1632227461' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cb9f7f48-76e0-4368-ab87-2a09a8487ccd"}]: dispatch 2024-02-21T23:43:40.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:39 smithi176 ceph-mon[47172]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cb9f7f48-76e0-4368-ab87-2a09a8487ccd"}]: dispatch 2024-02-21T23:43:40.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:39 smithi176 ceph-mon[47172]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cb9f7f48-76e0-4368-ab87-2a09a8487ccd"}]': finished 2024-02-21T23:43:40.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:39 smithi176 ceph-mon[47172]: osdmap e28: 5 total, 4 up, 5 in 2024-02-21T23:43:40.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:39 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-21T23:43:40.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:39 smithi176 ceph-mon[47172]: pgmap v81: 1 pgs: 1 remapped+peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-02-21T23:43:40.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:39 smithi176 ceph-mon[47172]: from='client.? 172.21.15.176:0/454335876' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-21T23:43:40.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:39 smithi178 ceph-mon[47547]: osdmap e27: 4 total, 4 up, 4 in 2024-02-21T23:43:40.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:39 smithi178 ceph-mon[47547]: from='client.? 172.21.15.176:0/1632227461' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cb9f7f48-76e0-4368-ab87-2a09a8487ccd"}]: dispatch 2024-02-21T23:43:40.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:39 smithi178 ceph-mon[47547]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cb9f7f48-76e0-4368-ab87-2a09a8487ccd"}]: dispatch 2024-02-21T23:43:40.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:39 smithi178 ceph-mon[47547]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cb9f7f48-76e0-4368-ab87-2a09a8487ccd"}]': finished 2024-02-21T23:43:40.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:39 smithi178 ceph-mon[47547]: osdmap e28: 5 total, 4 up, 5 in 2024-02-21T23:43:40.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:39 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-21T23:43:40.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:39 smithi178 ceph-mon[47547]: pgmap v81: 1 pgs: 1 remapped+peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-02-21T23:43:40.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:39 smithi178 ceph-mon[47547]: from='client.? 172.21.15.176:0/454335876' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-21T23:43:41.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:40 smithi178 ceph-mon[47547]: osdmap e29: 5 total, 4 up, 5 in 2024-02-21T23:43:41.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:40 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-21T23:43:41.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:40 smithi090 ceph-mon[43385]: osdmap e29: 5 total, 4 up, 5 in 2024-02-21T23:43:41.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:40 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-21T23:43:41.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:40 smithi176 ceph-mon[47172]: osdmap e29: 5 total, 4 up, 5 in 2024-02-21T23:43:41.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:40 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-21T23:43:42.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:41 smithi178 ceph-mon[47547]: pgmap v83: 1 pgs: 1 remapped+peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-02-21T23:43:42.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:41 smithi090 ceph-mon[43385]: pgmap v83: 1 pgs: 1 remapped+peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-02-21T23:43:42.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:41 smithi176 ceph-mon[47172]: pgmap v83: 1 pgs: 1 remapped+peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-02-21T23:43:44.673 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:44 smithi176 ceph-mon[47172]: pgmap v84: 1 pgs: 1 remapped+peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-02-21T23:43:44.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:44 smithi178 ceph-mon[47547]: pgmap v84: 1 pgs: 1 remapped+peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-02-21T23:43:44.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:44 smithi090 ceph-mon[43385]: pgmap v84: 1 pgs: 1 remapped+peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-02-21T23:43:45.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:45 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-21T23:43:45.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:45 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:45.793 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:45 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-21T23:43:45.794 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:45 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:45.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:45 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-21T23:43:45.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:45 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:46.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:46 smithi090 ceph-mon[43385]: Deploying daemon osd.4 on smithi176 2024-02-21T23:43:46.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:46 smithi090 ceph-mon[43385]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-02-21T23:43:46.742 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:46 smithi176 ceph-mon[47172]: Deploying daemon osd.4 on smithi176 2024-02-21T23:43:46.742 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:46 smithi176 ceph-mon[47172]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-02-21T23:43:46.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:46 smithi178 ceph-mon[47547]: Deploying daemon osd.4 on smithi176 2024-02-21T23:43:46.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:46 smithi178 ceph-mon[47547]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-02-21T23:43:47.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:47 smithi090 ceph-mon[43385]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-02-21T23:43:48.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:47 smithi176 ceph-mon[47172]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-02-21T23:43:48.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:47 smithi178 ceph-mon[47547]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-02-21T23:43:49.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:49 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:49.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:49 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:43:49.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:49 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:49.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:49 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:49.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:49 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:43:49.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:49 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:49.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:49 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:49.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:49 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:43:49.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:49 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:50.533 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:50.549+0000 7fcf8f7fe700 1 -- 172.21.15.176:0/2746837197 <== mgr.14150 v2:172.21.15.90:6800/3697394685 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fcfa0068df0 con 0x7fcf880a1120 2024-02-21T23:43:50.534 INFO:teuthology.orchestra.run.smithi176.stdout:Created osd(s) 4 on host 'smithi176' 2024-02-21T23:43:50.536 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:50.551+0000 7fcfa4da2700 1 -- 172.21.15.176:0/2746837197 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fcf880a1120 msgr2=0x7fcf880a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:43:50.536 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:50.551+0000 7fcfa4da2700 1 --2- 172.21.15.176:0/2746837197 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fcf880a1120 0x7fcf880a35e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcf94005950 tx=0x7fcf940058e0 comp rx=0 tx=0).stop 2024-02-21T23:43:50.536 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:50.551+0000 7fcfa4da2700 1 -- 172.21.15.176:0/2746837197 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfa0101830 msgr2=0x7fcfa019e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:43:50.536 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:50.551+0000 7fcfa4da2700 1 --2- 172.21.15.176:0/2746837197 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfa0101830 0x7fcfa019e210 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fcf9800ada0 tx=0x7fcf9800c330 comp rx=0 tx=0).stop 2024-02-21T23:43:50.536 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:50.551+0000 7fcfa4da2700 1 -- 172.21.15.176:0/2746837197 shutdown_connections 2024-02-21T23:43:50.536 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:50.551+0000 7fcfa4da2700 1 --2- 172.21.15.176:0/2746837197 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fcf880a1120 0x7fcf880a35e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:43:50.537 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:50.551+0000 7fcfa4da2700 1 --2- 172.21.15.176:0/2746837197 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfa0101830 0x7fcfa019e210 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:43:50.537 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:50.551+0000 7fcfa4da2700 1 --2- 172.21.15.176:0/2746837197 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcfa00ffc90 0x7fcfa0196d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:43:50.537 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:50.551+0000 7fcfa4da2700 1 --2- 172.21.15.176:0/2746837197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfa0100e90 0x7fcfa0197240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:43:50.537 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:50.551+0000 7fcfa4da2700 1 -- 172.21.15.176:0/2746837197 >> 172.21.15.176:0/2746837197 conn(0x7fcfa00fb230 msgr2=0x7fcfa00686e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:43:50.537 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:50.551+0000 7fcfa4da2700 1 -- 172.21.15.176:0/2746837197 shutdown_connections 2024-02-21T23:43:50.537 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:43:50.551+0000 7fcfa4da2700 1 -- 172.21.15.176:0/2746837197 wait complete. 2024-02-21T23:43:50.810 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:50 smithi176 ceph-mon[47172]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-21T23:43:50.811 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:50 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:50.811 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:50 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:50.811 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:50 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:50.811 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:50 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:43:50.811 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:50 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:50.811 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:50 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:50.811 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:50 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:50.906 DEBUG:teuthology.orchestra.run.smithi176:osd.4> sudo journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@osd.4.service 2024-02-21T23:43:50.908 INFO:tasks.cephadm:Deploying osd.5 on smithi178 with /dev/vg_nvme/lv_3... 2024-02-21T23:43:50.908 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-21T23:43:50.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:50 smithi178 ceph-mon[47547]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-21T23:43:50.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:50 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:50.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:50 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:50.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:50 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:50.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:50 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:43:50.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:50 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:50.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:50 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:50.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:50 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:50.938 INFO:journalctl@ceph.osd.4.smithi176.stdout:-- Logs begin at Wed 2024-02-21 23:23:58 UTC. -- 2024-02-21T23:43:50.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:50 smithi090 ceph-mon[43385]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-21T23:43:50.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:50 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:50.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:50 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:50.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:50 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:50.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:50 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:43:50.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:50 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:50.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:50 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:50.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:50 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:51.140 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.c/config 2024-02-21T23:43:51.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:51 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:43:51.860 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:51 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:43:51.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:51 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:43:52.601 INFO:teuthology.orchestra.run.smithi178.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-21T23:43:52.601 INFO:teuthology.orchestra.run.smithi178.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-21T23:43:52.601 INFO:teuthology.orchestra.run.smithi178.stderr: stderr: 10+0 records in 2024-02-21T23:43:52.601 INFO:teuthology.orchestra.run.smithi178.stderr:10+0 records out 2024-02-21T23:43:52.602 INFO:teuthology.orchestra.run.smithi178.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00827864 s, 1.3 GB/s 2024-02-21T23:43:52.602 INFO:teuthology.orchestra.run.smithi178.stderr:--> Zapping successful for: 2024-02-21T23:43:52.668 INFO:journalctl@ceph.osd.4.smithi176.stdout:Feb 21 23:43:52 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-4[60843]: 2024-02-21T23:43:52.396+0000 7f6132619640 -1 osd.4 0 log_to_monitors true 2024-02-21T23:43:52.668 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:52 smithi176 ceph-mon[47172]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-02-21T23:43:52.668 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:52 smithi176 ceph-mon[47172]: from='osd.4 [v2:172.21.15.176:6808/3548915687,v1:172.21.15.176:6809/3548915687]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-21T23:43:52.668 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:52 smithi176 ceph-mon[47172]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-21T23:43:52.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:52 smithi178 ceph-mon[47547]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-02-21T23:43:52.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:52 smithi178 ceph-mon[47547]: from='osd.4 [v2:172.21.15.176:6808/3548915687,v1:172.21.15.176:6809/3548915687]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-21T23:43:52.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:52 smithi178 ceph-mon[47547]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-21T23:43:52.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:52 smithi090 ceph-mon[43385]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-02-21T23:43:52.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:52 smithi090 ceph-mon[43385]: from='osd.4 [v2:172.21.15.176:6808/3548915687,v1:172.21.15.176:6809/3548915687]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-21T23:43:52.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:52 smithi090 ceph-mon[43385]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-21T23:43:53.002 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph orch daemon add osd smithi178:vg_nvme/lv_3 2024-02-21T23:43:53.259 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.c/config 2024-02-21T23:43:53.976 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:53 smithi178 ceph-mon[47547]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-21T23:43:53.976 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:53 smithi178 ceph-mon[47547]: osdmap e30: 5 total, 4 up, 5 in 2024-02-21T23:43:53.976 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:53 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-21T23:43:53.976 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:53 smithi178 ceph-mon[47547]: from='osd.4 [v2:172.21.15.176:6808/3548915687,v1:172.21.15.176:6809/3548915687]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-21T23:43:53.976 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:53 smithi178 ceph-mon[47547]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-21T23:43:53.977 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:53 smithi178 ceph-mon[47547]: Detected new or changed devices on smithi176 2024-02-21T23:43:53.977 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:53 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:53.977 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:53 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:53.977 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:53 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:43:53.977 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:53 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:43:53.977 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:53 smithi178 ceph-mon[47547]: Adjusting osd_memory_target on smithi176 to 8557M 2024-02-21T23:43:53.977 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:53 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:53.977 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:53 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:53.977 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:53 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:43:53.977 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:53 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:53.978 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:53 smithi178 ceph-mon[47547]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-21T23:43:53.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:53 smithi090 ceph-mon[43385]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-21T23:43:53.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:53 smithi090 ceph-mon[43385]: osdmap e30: 5 total, 4 up, 5 in 2024-02-21T23:43:53.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:53 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-21T23:43:53.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:53 smithi090 ceph-mon[43385]: from='osd.4 [v2:172.21.15.176:6808/3548915687,v1:172.21.15.176:6809/3548915687]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-21T23:43:53.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:53 smithi090 ceph-mon[43385]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-21T23:43:53.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:53 smithi090 ceph-mon[43385]: Detected new or changed devices on smithi176 2024-02-21T23:43:53.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:53 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:53.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:53 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:53.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:53 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:43:53.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:53 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:43:53.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:53 smithi090 ceph-mon[43385]: Adjusting osd_memory_target on smithi176 to 8557M 2024-02-21T23:43:53.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:53 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:53.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:53 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:53.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:53 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:43:53.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:53 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:53.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:53 smithi090 ceph-mon[43385]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-21T23:43:54.084 INFO:journalctl@ceph.osd.4.smithi176.stdout:Feb 21 23:43:53 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-4[60843]: 2024-02-21T23:43:53.739+0000 7f6127c7d700 -1 osd.4 0 waiting for initial osdmap 2024-02-21T23:43:54.084 INFO:journalctl@ceph.osd.4.smithi176.stdout:Feb 21 23:43:53 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-4[60843]: 2024-02-21T23:43:53.747+0000 7f611fa6a700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-21T23:43:54.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:53 smithi176 ceph-mon[47172]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-21T23:43:54.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:53 smithi176 ceph-mon[47172]: osdmap e30: 5 total, 4 up, 5 in 2024-02-21T23:43:54.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:53 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-21T23:43:54.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:53 smithi176 ceph-mon[47172]: from='osd.4 [v2:172.21.15.176:6808/3548915687,v1:172.21.15.176:6809/3548915687]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-21T23:43:54.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:53 smithi176 ceph-mon[47172]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-21T23:43:54.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:53 smithi176 ceph-mon[47172]: Detected new or changed devices on smithi176 2024-02-21T23:43:54.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:53 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:54.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:53 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:54.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:53 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:43:54.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:53 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:43:54.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:53 smithi176 ceph-mon[47172]: Adjusting osd_memory_target on smithi176 to 8557M 2024-02-21T23:43:54.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:53 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:54.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:53 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:54.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:53 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:43:54.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:53 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:43:54.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:53 smithi176 ceph-mon[47172]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-21T23:43:54.362 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.375+0000 7f7a5d622700 1 -- 172.21.15.178:0/1991453466 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a58104150 msgr2=0x7f7a58106540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:43:54.362 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.375+0000 7f7a5d622700 1 --2- 172.21.15.178:0/1991453466 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a58104150 0x7f7a58106540 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7a48009990 tx=0x7f7a48009ca0 comp rx=0 tx=0).stop 2024-02-21T23:43:54.362 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.375+0000 7f7a5d622700 1 -- 172.21.15.178:0/1991453466 shutdown_connections 2024-02-21T23:43:54.362 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.375+0000 7f7a5d622700 1 --2- 172.21.15.178:0/1991453466 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7a58106a80 0x7f7a58108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:43:54.362 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.375+0000 7f7a5d622700 1 --2- 172.21.15.178:0/1991453466 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a58101820 0x7f7a58103c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:43:54.362 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.375+0000 7f7a5d622700 1 --2- 172.21.15.178:0/1991453466 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a58104150 0x7f7a58106540 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:43:54.362 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.375+0000 7f7a5d622700 1 -- 172.21.15.178:0/1991453466 >> 172.21.15.178:0/1991453466 conn(0x7f7a580fb230 msgr2=0x7f7a580fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:43:54.362 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.375+0000 7f7a5d622700 1 -- 172.21.15.178:0/1991453466 shutdown_connections 2024-02-21T23:43:54.363 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.375+0000 7f7a5d622700 1 -- 172.21.15.178:0/1991453466 wait complete. 2024-02-21T23:43:54.363 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.376+0000 7f7a5d622700 1 Processor -- start 2024-02-21T23:43:54.363 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.376+0000 7f7a5d622700 1 -- start start 2024-02-21T23:43:54.363 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.376+0000 7f7a5d622700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a58101820 0x7f7a58196cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:43:54.363 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.376+0000 7f7a5d622700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7a58104150 0x7f7a581971f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:43:54.363 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.376+0000 7f7a5d622700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a58106a80 0x7f7a5819e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:43:54.363 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.376+0000 7f7a5d622700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5819e7e0 con 0x7f7a58106a80 2024-02-21T23:43:54.363 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.376+0000 7f7a5d622700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5819e920 con 0x7f7a58104150 2024-02-21T23:43:54.363 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.376+0000 7f7a5d622700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5819eae0 con 0x7f7a58101820 2024-02-21T23:43:54.363 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.376+0000 7f7a56ffd700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a58101820 0x7f7a58196cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:43:54.363 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.376+0000 7f7a567fc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7a58104150 0x7f7a581971f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:43:54.363 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.376+0000 7f7a56ffd700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a58101820 0x7f7a58196cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:40982/0 (socket says 172.21.15.178:40982) 2024-02-21T23:43:54.363 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.376+0000 7f7a56ffd700 1 -- 172.21.15.178:0/1407180878 learned_addr learned my addr 172.21.15.178:0/1407180878 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-21T23:43:54.364 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.376+0000 7f7a577fe700 1 --2- 172.21.15.178:0/1407180878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a58106a80 0x7f7a5819e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:43:54.364 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.377+0000 7f7a56ffd700 1 -- 172.21.15.178:0/1407180878 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7a58104150 msgr2=0x7f7a581971f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:43:54.364 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.377+0000 7f7a56ffd700 1 --2- 172.21.15.178:0/1407180878 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7a58104150 0x7f7a581971f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:43:54.364 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.377+0000 7f7a56ffd700 1 -- 172.21.15.178:0/1407180878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a58106a80 msgr2=0x7f7a5819e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:43:54.364 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.377+0000 7f7a56ffd700 1 --2- 172.21.15.178:0/1407180878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a58106a80 0x7f7a5819e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:43:54.364 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.377+0000 7f7a56ffd700 1 -- 172.21.15.178:0/1407180878 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a480095f0 con 0x7f7a58101820 2024-02-21T23:43:54.364 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.377+0000 7f7a577fe700 1 --2- 172.21.15.178:0/1407180878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a58106a80 0x7f7a5819e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:43:54.364 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.377+0000 7f7a56ffd700 1 --2- 172.21.15.178:0/1407180878 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a58101820 0x7f7a58196cb0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c00d950 tx=0x7f7a4c00dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:43:54.364 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.377+0000 7f7a3ffff700 1 -- 172.21.15.178:0/1407180878 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c009940 con 0x7f7a58101820 2024-02-21T23:43:54.364 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.377+0000 7f7a3ffff700 1 -- 172.21.15.178:0/1407180878 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7a4c010460 con 0x7f7a58101820 2024-02-21T23:43:54.364 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.377+0000 7f7a3ffff700 1 -- 172.21.15.178:0/1407180878 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c00f620 con 0x7f7a58101820 2024-02-21T23:43:54.365 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.377+0000 7f7a5d622700 1 -- 172.21.15.178:0/1407180878 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a5819edc0 con 0x7f7a58101820 2024-02-21T23:43:54.365 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.377+0000 7f7a5d622700 1 -- 172.21.15.178:0/1407180878 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a5819f1b0 con 0x7f7a58101820 2024-02-21T23:43:54.365 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.378+0000 7f7a5d622700 1 -- 172.21.15.178:0/1407180878 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a580fcdb0 con 0x7f7a58101820 2024-02-21T23:43:54.366 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.379+0000 7f7a3ffff700 1 -- 172.21.15.178:0/1407180878 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7f7a4c0108d0 con 0x7f7a58101820 2024-02-21T23:43:54.366 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.379+0000 7f7a3ffff700 1 --2- 172.21.15.178:0/1407180878 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f7a400a1220 0x7f7a400a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:43:54.366 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.379+0000 7f7a3ffff700 1 -- 172.21.15.178:0/1407180878 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 3821+0+0 (secure 0 0 0) 0x7f7a4c0cb5f0 con 0x7f7a58101820 2024-02-21T23:43:54.366 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.380+0000 7f7a567fc700 1 --2- 172.21.15.178:0/1407180878 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f7a400a1220 0x7f7a400a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:43:54.366 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.380+0000 7f7a567fc700 1 --2- 172.21.15.178:0/1407180878 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f7a400a1220 0x7f7a400a36e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7a58198170 tx=0x7f7a48003790 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:43:54.368 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.384+0000 7f7a3ffff700 1 -- 172.21.15.178:0/1407180878 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f7a4c095e70 con 0x7f7a58101820 2024-02-21T23:43:54.566 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:43:54.582+0000 7f7a5d622700 1 -- 172.21.15.178:0/1407180878 --> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f7a580611b0 con 0x7f7a400a1220 2024-02-21T23:43:54.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:54 smithi090 ceph-mon[43385]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-02-21T23:43:54.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:54 smithi090 ceph-mon[43385]: osdmap e31: 5 total, 4 up, 5 in 2024-02-21T23:43:54.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:54 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-21T23:43:54.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:54 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-21T23:43:54.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:54 smithi090 ceph-mon[43385]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:43:54.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:54 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-21T23:43:54.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:54 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-21T23:43:54.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:54 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:54.978 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:54 smithi178 ceph-mon[47547]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-02-21T23:43:54.978 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:54 smithi178 ceph-mon[47547]: osdmap e31: 5 total, 4 up, 5 in 2024-02-21T23:43:54.978 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:54 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-21T23:43:54.978 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:54 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-21T23:43:54.978 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:54 smithi178 ceph-mon[47547]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:43:54.979 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:54 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-21T23:43:54.979 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:54 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-21T23:43:54.979 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:54 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:55.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:54 smithi176 ceph-mon[47172]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-02-21T23:43:55.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:54 smithi176 ceph-mon[47172]: osdmap e31: 5 total, 4 up, 5 in 2024-02-21T23:43:55.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:54 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-21T23:43:55.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:54 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-21T23:43:55.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:54 smithi176 ceph-mon[47172]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:43:55.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:54 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-21T23:43:55.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:54 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-21T23:43:55.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:54 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:43:56.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:55 smithi176 ceph-mon[47172]: purged_snaps scrub starts 2024-02-21T23:43:56.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:55 smithi176 ceph-mon[47172]: purged_snaps scrub ok 2024-02-21T23:43:56.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:55 smithi176 ceph-mon[47172]: osd.4 [v2:172.21.15.176:6808/3548915687,v1:172.21.15.176:6809/3548915687] boot 2024-02-21T23:43:56.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:55 smithi176 ceph-mon[47172]: osdmap e32: 5 total, 5 up, 5 in 2024-02-21T23:43:56.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:55 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-21T23:43:56.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:55 smithi176 ceph-mon[47172]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-02-21T23:43:56.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:55 smithi178 ceph-mon[47547]: purged_snaps scrub starts 2024-02-21T23:43:56.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:55 smithi178 ceph-mon[47547]: purged_snaps scrub ok 2024-02-21T23:43:56.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:55 smithi178 ceph-mon[47547]: osd.4 [v2:172.21.15.176:6808/3548915687,v1:172.21.15.176:6809/3548915687] boot 2024-02-21T23:43:56.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:55 smithi178 ceph-mon[47547]: osdmap e32: 5 total, 5 up, 5 in 2024-02-21T23:43:56.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:55 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-21T23:43:56.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:55 smithi178 ceph-mon[47547]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-02-21T23:43:56.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:55 smithi090 ceph-mon[43385]: purged_snaps scrub starts 2024-02-21T23:43:56.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:55 smithi090 ceph-mon[43385]: purged_snaps scrub ok 2024-02-21T23:43:56.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:55 smithi090 ceph-mon[43385]: osd.4 [v2:172.21.15.176:6808/3548915687,v1:172.21.15.176:6809/3548915687] boot 2024-02-21T23:43:56.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:55 smithi090 ceph-mon[43385]: osdmap e32: 5 total, 5 up, 5 in 2024-02-21T23:43:56.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:55 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-21T23:43:56.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:55 smithi090 ceph-mon[43385]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-02-21T23:43:57.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:56 smithi176 ceph-mon[47172]: osdmap e33: 5 total, 5 up, 5 in 2024-02-21T23:43:57.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:56 smithi176 ceph-mon[47172]: from='client.? 172.21.15.178:0/2372541675' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54749022-c50d-4831-a1d6-8412b59c8a01"}]: dispatch 2024-02-21T23:43:57.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:56 smithi176 ceph-mon[47172]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54749022-c50d-4831-a1d6-8412b59c8a01"}]: dispatch 2024-02-21T23:43:57.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:56 smithi176 ceph-mon[47172]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "54749022-c50d-4831-a1d6-8412b59c8a01"}]': finished 2024-02-21T23:43:57.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:56 smithi176 ceph-mon[47172]: osdmap e34: 6 total, 5 up, 6 in 2024-02-21T23:43:57.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:56 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-21T23:43:57.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:56 smithi178 ceph-mon[47547]: osdmap e33: 5 total, 5 up, 5 in 2024-02-21T23:43:57.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:56 smithi178 ceph-mon[47547]: from='client.? 172.21.15.178:0/2372541675' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54749022-c50d-4831-a1d6-8412b59c8a01"}]: dispatch 2024-02-21T23:43:57.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:56 smithi178 ceph-mon[47547]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54749022-c50d-4831-a1d6-8412b59c8a01"}]: dispatch 2024-02-21T23:43:57.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:56 smithi178 ceph-mon[47547]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "54749022-c50d-4831-a1d6-8412b59c8a01"}]': finished 2024-02-21T23:43:57.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:56 smithi178 ceph-mon[47547]: osdmap e34: 6 total, 5 up, 6 in 2024-02-21T23:43:57.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:56 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-21T23:43:57.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:56 smithi090 ceph-mon[43385]: osdmap e33: 5 total, 5 up, 5 in 2024-02-21T23:43:57.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:56 smithi090 ceph-mon[43385]: from='client.? 172.21.15.178:0/2372541675' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54749022-c50d-4831-a1d6-8412b59c8a01"}]: dispatch 2024-02-21T23:43:57.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:56 smithi090 ceph-mon[43385]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54749022-c50d-4831-a1d6-8412b59c8a01"}]: dispatch 2024-02-21T23:43:57.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:56 smithi090 ceph-mon[43385]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "54749022-c50d-4831-a1d6-8412b59c8a01"}]': finished 2024-02-21T23:43:57.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:56 smithi090 ceph-mon[43385]: osdmap e34: 6 total, 5 up, 6 in 2024-02-21T23:43:57.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:56 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-21T23:43:58.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:57 smithi176 ceph-mon[47172]: from='client.? 172.21.15.178:0/2341535087' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-21T23:43:58.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:43:57 smithi176 ceph-mon[47172]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-02-21T23:43:58.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:57 smithi178 ceph-mon[47547]: from='client.? 172.21.15.178:0/2341535087' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-21T23:43:58.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:43:57 smithi178 ceph-mon[47547]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-02-21T23:43:58.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:57 smithi090 ceph-mon[43385]: from='client.? 172.21.15.178:0/2341535087' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-21T23:43:58.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:43:57 smithi090 ceph-mon[43385]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-02-21T23:44:00.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:00 smithi090 ceph-mon[43385]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-02-21T23:44:00.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:00 smithi176 ceph-mon[47172]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-02-21T23:44:00.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:00 smithi178 ceph-mon[47547]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-02-21T23:44:02.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:02 smithi090 ceph-mon[43385]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-02-21T23:44:02.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:02 smithi176 ceph-mon[47172]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-02-21T23:44:02.868 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:02 smithi178 ceph-mon[47547]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-02-21T23:44:03.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:03 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-21T23:44:03.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:03 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:03.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:03 smithi178 ceph-mon[47547]: Deploying daemon osd.5 on smithi178 2024-02-21T23:44:03.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:03 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-21T23:44:03.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:03 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:03.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:03 smithi090 ceph-mon[43385]: Deploying daemon osd.5 on smithi178 2024-02-21T23:44:03.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:03 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-21T23:44:03.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:03 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:03.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:03 smithi176 ceph-mon[47172]: Deploying daemon osd.5 on smithi178 2024-02-21T23:44:04.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:04 smithi090 ceph-mon[43385]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-02-21T23:44:04.737 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:04 smithi178 ceph-mon[47547]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-02-21T23:44:04.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:04 smithi176 ceph-mon[47172]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-02-21T23:44:06.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:06 smithi178 ceph-mon[47547]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-02-21T23:44:06.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:06 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:06.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:06 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:44:06.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:06 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:06.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:06 smithi090 ceph-mon[43385]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-02-21T23:44:06.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:06 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:06.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:06 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:44:06.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:06 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:06.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:06 smithi176 ceph-mon[47172]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-02-21T23:44:06.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:06 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:06.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:06 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:44:06.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:06 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:07.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:07 smithi090 ceph-mon[43385]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-02-21T23:44:08.006 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:07 smithi178 ceph-mon[47547]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-02-21T23:44:08.083 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:07 smithi176 ceph-mon[47172]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-02-21T23:44:09.178 INFO:teuthology.orchestra.run.smithi178.stdout:Created osd(s) 5 on host 'smithi178' 2024-02-21T23:44:09.178 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:09.192+0000 7f7a3ffff700 1 -- 172.21.15.178:0/1407180878 <== mgr.14150 v2:172.21.15.90:6800/3697394685 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7a580611b0 con 0x7f7a400a1220 2024-02-21T23:44:09.180 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:09.195+0000 7f7a5d622700 1 -- 172.21.15.178:0/1407180878 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f7a400a1220 msgr2=0x7f7a400a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:44:09.180 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:09.195+0000 7f7a5d622700 1 --2- 172.21.15.178:0/1407180878 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f7a400a1220 0x7f7a400a36e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7a58198170 tx=0x7f7a48003790 comp rx=0 tx=0).stop 2024-02-21T23:44:09.180 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:09.195+0000 7f7a5d622700 1 -- 172.21.15.178:0/1407180878 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a58101820 msgr2=0x7f7a58196cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:44:09.180 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:09.195+0000 7f7a5d622700 1 --2- 172.21.15.178:0/1407180878 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a58101820 0x7f7a58196cb0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c00d950 tx=0x7f7a4c00dc60 comp rx=0 tx=0).stop 2024-02-21T23:44:09.180 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:09.195+0000 7f7a5d622700 1 -- 172.21.15.178:0/1407180878 shutdown_connections 2024-02-21T23:44:09.180 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:09.195+0000 7f7a5d622700 1 --2- 172.21.15.178:0/1407180878 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f7a400a1220 0x7f7a400a36e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:44:09.180 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:09.195+0000 7f7a5d622700 1 --2- 172.21.15.178:0/1407180878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a58106a80 0x7f7a5819e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:44:09.180 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:09.195+0000 7f7a5d622700 1 --2- 172.21.15.178:0/1407180878 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a58101820 0x7f7a58196cb0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:44:09.180 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:09.195+0000 7f7a5d622700 1 --2- 172.21.15.178:0/1407180878 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7a58104150 0x7f7a581971f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:44:09.181 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:09.195+0000 7f7a5d622700 1 -- 172.21.15.178:0/1407180878 >> 172.21.15.178:0/1407180878 conn(0x7f7a580fb230 msgr2=0x7f7a581036e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:44:09.181 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:09.195+0000 7f7a5d622700 1 -- 172.21.15.178:0/1407180878 shutdown_connections 2024-02-21T23:44:09.181 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:09.195+0000 7f7a5d622700 1 -- 172.21.15.178:0/1407180878 wait complete. 2024-02-21T23:44:09.572 DEBUG:teuthology.orchestra.run.smithi178:osd.5> sudo journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@osd.5.service 2024-02-21T23:44:09.575 INFO:tasks.cephadm:Deploying osd.6 on smithi090 with /dev/vg_nvme/lv_2... 2024-02-21T23:44:09.576 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-21T23:44:09.613 INFO:journalctl@ceph.osd.5.smithi178.stdout:-- Logs begin at Wed 2024-02-21 23:24:26 UTC. -- 2024-02-21T23:44:09.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:09 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:09.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:09 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:09.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:09 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:09.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:09 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:44:09.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:09 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:09.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:09 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:09.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:09 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:09.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:09 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:44:09.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:09 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:09.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:09 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:09.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:09 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:09.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:09 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:44:09.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:09 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:09.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:09 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:09.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:09 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:09.729 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:09 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:44:09.807 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.a/config 2024-02-21T23:44:09.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:09 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:09.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:09 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:09.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:09 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:09.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:09 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:44:09.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:09 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:09.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:09 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:09.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:09 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:09.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:09 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:44:10.379 INFO:journalctl@ceph.osd.5.smithi178.stdout:Feb 21 23:44:10 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-5[61968]: 2024-02-21T23:44:10.315+0000 7f9ad929d640 -1 osd.5 0 log_to_monitors true 2024-02-21T23:44:10.585 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:10 smithi090 ceph-mon[43385]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-02-21T23:44:10.585 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:10 smithi090 ceph-mon[43385]: from='osd.5 [v2:172.21.15.178:6808/286274518,v1:172.21.15.178:6809/286274518]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-21T23:44:10.585 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:10 smithi090 ceph-mon[43385]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-21T23:44:10.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:10 smithi178 ceph-mon[47547]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-02-21T23:44:10.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:10 smithi178 ceph-mon[47547]: from='osd.5 [v2:172.21.15.178:6808/286274518,v1:172.21.15.178:6809/286274518]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-21T23:44:10.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:10 smithi178 ceph-mon[47547]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-21T23:44:10.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:10 smithi176 ceph-mon[47172]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-02-21T23:44:10.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:10 smithi176 ceph-mon[47172]: from='osd.5 [v2:172.21.15.178:6808/286274518,v1:172.21.15.178:6809/286274518]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-21T23:44:10.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:10 smithi176 ceph-mon[47172]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-21T23:44:11.104 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-21T23:44:11.104 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-21T23:44:11.104 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10+0 records in 2024-02-21T23:44:11.104 INFO:teuthology.orchestra.run.smithi090.stderr:10+0 records out 2024-02-21T23:44:11.104 INFO:teuthology.orchestra.run.smithi090.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113031 s, 928 MB/s 2024-02-21T23:44:11.104 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping successful for: 2024-02-21T23:44:11.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:11 smithi090 ceph-mon[43385]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-21T23:44:11.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:11 smithi090 ceph-mon[43385]: osdmap e35: 6 total, 5 up, 6 in 2024-02-21T23:44:11.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:11 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-21T23:44:11.445 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:11 smithi090 ceph-mon[43385]: from='osd.5 [v2:172.21.15.178:6808/286274518,v1:172.21.15.178:6809/286274518]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-02-21T23:44:11.446 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:11 smithi090 ceph-mon[43385]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-02-21T23:44:11.522 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph orch daemon add osd smithi090:vg_nvme/lv_2 2024-02-21T23:44:11.674 INFO:journalctl@ceph.osd.5.smithi178.stdout:Feb 21 23:44:11 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-5[61968]: 2024-02-21T23:44:11.406+0000 7f9ace901700 -1 osd.5 0 waiting for initial osdmap 2024-02-21T23:44:11.674 INFO:journalctl@ceph.osd.5.smithi178.stdout:Feb 21 23:44:11 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-5[61968]: 2024-02-21T23:44:11.414+0000 7f9ac9ef5700 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-21T23:44:11.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:11 smithi178 ceph-mon[47547]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-21T23:44:11.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:11 smithi178 ceph-mon[47547]: osdmap e35: 6 total, 5 up, 6 in 2024-02-21T23:44:11.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:11 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-21T23:44:11.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:11 smithi178 ceph-mon[47547]: from='osd.5 [v2:172.21.15.178:6808/286274518,v1:172.21.15.178:6809/286274518]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-02-21T23:44:11.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:11 smithi178 ceph-mon[47547]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-02-21T23:44:11.743 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.a/config 2024-02-21T23:44:11.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:11 smithi176 ceph-mon[47172]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-21T23:44:11.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:11 smithi176 ceph-mon[47172]: osdmap e35: 6 total, 5 up, 6 in 2024-02-21T23:44:11.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:11 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-21T23:44:11.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:11 smithi176 ceph-mon[47172]: from='osd.5 [v2:172.21.15.178:6808/286274518,v1:172.21.15.178:6809/286274518]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-02-21T23:44:11.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:11 smithi176 ceph-mon[47172]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-02-21T23:44:12.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:12 smithi090 ceph-mon[43385]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-02-21T23:44:12.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:12 smithi090 ceph-mon[43385]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-02-21T23:44:12.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:12 smithi090 ceph-mon[43385]: osdmap e36: 6 total, 5 up, 6 in 2024-02-21T23:44:12.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:12 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-21T23:44:12.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:12 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-21T23:44:12.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:12 smithi090 ceph-mon[43385]: osd.5 [v2:172.21.15.178:6808/286274518,v1:172.21.15.178:6809/286274518] boot 2024-02-21T23:44:12.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:12 smithi090 ceph-mon[43385]: osdmap e37: 6 total, 6 up, 6 in 2024-02-21T23:44:12.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:12 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-21T23:44:12.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:12 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:12.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:12 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:12.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:12 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:44:12.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:12 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:44:12.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:12 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:12.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:12 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:12.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:12 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:44:12.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:12 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:12.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:12 smithi178 ceph-mon[47547]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-02-21T23:44:12.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:12 smithi178 ceph-mon[47547]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-02-21T23:44:12.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:12 smithi178 ceph-mon[47547]: osdmap e36: 6 total, 5 up, 6 in 2024-02-21T23:44:12.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:12 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-21T23:44:12.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:12 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-21T23:44:12.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:12 smithi178 ceph-mon[47547]: osd.5 [v2:172.21.15.178:6808/286274518,v1:172.21.15.178:6809/286274518] boot 2024-02-21T23:44:12.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:12 smithi178 ceph-mon[47547]: osdmap e37: 6 total, 6 up, 6 in 2024-02-21T23:44:12.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:12 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-21T23:44:12.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:12 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:12.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:12 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:12.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:12 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:44:12.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:12 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:44:12.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:12 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:12.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:12 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:12.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:12 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:44:12.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:12 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:12.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.737+0000 7f92197d1700 1 -- 172.21.15.90:0/2673733960 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92140ff400 msgr2=0x7f92140ff820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:44:12.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.737+0000 7f92197d1700 1 --2- 172.21.15.90:0/2673733960 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92140ff400 0x7f92140ff820 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f920800aad0 tx=0x7f920800ade0 comp rx=0 tx=0).stop 2024-02-21T23:44:12.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.737+0000 7f92197d1700 1 -- 172.21.15.90:0/2673733960 shutdown_connections 2024-02-21T23:44:12.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.737+0000 7f92197d1700 1 --2- 172.21.15.90:0/2673733960 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9214100fa0 0x7f9214107850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:44:12.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.737+0000 7f92197d1700 1 --2- 172.21.15.90:0/2673733960 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92140ff400 0x7f92140ff820 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:44:12.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.737+0000 7f92197d1700 1 --2- 172.21.15.90:0/2673733960 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9214100600 0x7f9214100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:44:12.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.737+0000 7f92197d1700 1 -- 172.21.15.90:0/2673733960 >> 172.21.15.90:0/2673733960 conn(0x7f92140fa9c0 msgr2=0x7f92140fcde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:44:12.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.737+0000 7f92197d1700 1 -- 172.21.15.90:0/2673733960 shutdown_connections 2024-02-21T23:44:12.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.737+0000 7f92197d1700 1 -- 172.21.15.90:0/2673733960 wait complete. 2024-02-21T23:44:12.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.738+0000 7f92197d1700 1 Processor -- start 2024-02-21T23:44:12.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.738+0000 7f92197d1700 1 -- start start 2024-02-21T23:44:12.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.738+0000 7f92197d1700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f92140ff400 0x7f9214103340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:44:12.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.738+0000 7f92197d1700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9214100600 0x7f92141019e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:44:12.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.738+0000 7f92197d1700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9214100fa0 0x7f9214101f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:44:12.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.738+0000 7f92197d1700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92141026c0 con 0x7f9214100600 2024-02-21T23:44:12.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.738+0000 7f92197d1700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9214102830 con 0x7f92140ff400 2024-02-21T23:44:12.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.738+0000 7f92197d1700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92141a0d70 con 0x7f9214100fa0 2024-02-21T23:44:12.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.738+0000 7f92127fc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9214100600 0x7f92141019e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:44:12.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.738+0000 7f92127fc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9214100600 0x7f92141019e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60730/0 (socket says 172.21.15.90:60730) 2024-02-21T23:44:12.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.738+0000 7f92127fc700 1 -- 172.21.15.90:0/472268007 learned_addr learned my addr 172.21.15.90:0/472268007 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:44:12.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.738+0000 7f9212ffd700 1 --2- 172.21.15.90:0/472268007 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f92140ff400 0x7f9214103340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:44:12.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.738+0000 7f92137fe700 1 --2- 172.21.15.90:0/472268007 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9214100fa0 0x7f9214101f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:44:12.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.738+0000 7f92127fc700 1 -- 172.21.15.90:0/472268007 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9214100fa0 msgr2=0x7f9214101f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:44:12.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.738+0000 7f92127fc700 1 --2- 172.21.15.90:0/472268007 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9214100fa0 0x7f9214101f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:44:12.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.738+0000 7f92127fc700 1 -- 172.21.15.90:0/472268007 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f92140ff400 msgr2=0x7f9214103340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:44:12.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.738+0000 7f92127fc700 1 --2- 172.21.15.90:0/472268007 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f92140ff400 0x7f9214103340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:44:12.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.738+0000 7f92127fc700 1 -- 172.21.15.90:0/472268007 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f920800a700 con 0x7f9214100600 2024-02-21T23:44:12.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.738+0000 7f9212ffd700 1 --2- 172.21.15.90:0/472268007 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f92140ff400 0x7f9214103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:44:12.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.738+0000 7f92137fe700 1 --2- 172.21.15.90:0/472268007 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9214100fa0 0x7f9214101f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:44:12.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.739+0000 7f92127fc700 1 --2- 172.21.15.90:0/472268007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9214100600 0x7f92141019e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f920400c920 tx=0x7f920400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:44:12.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.739+0000 7f91fbfff700 1 -- 172.21.15.90:0/472268007 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9204007b00 con 0x7f9214100600 2024-02-21T23:44:12.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.739+0000 7f91fbfff700 1 -- 172.21.15.90:0/472268007 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9204004cf0 con 0x7f9214100600 2024-02-21T23:44:12.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.739+0000 7f92197d1700 1 -- 172.21.15.90:0/472268007 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92141a0fa0 con 0x7f9214100600 2024-02-21T23:44:12.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.739+0000 7f91fbfff700 1 -- 172.21.15.90:0/472268007 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9204005610 con 0x7f9214100600 2024-02-21T23:44:12.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.739+0000 7f92197d1700 1 -- 172.21.15.90:0/472268007 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92141a14c0 con 0x7f9214100600 2024-02-21T23:44:12.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.740+0000 7f92197d1700 1 -- 172.21.15.90:0/472268007 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f921404ea40 con 0x7f9214100600 2024-02-21T23:44:12.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.741+0000 7f91fbfff700 1 -- 172.21.15.90:0/472268007 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7f9204004750 con 0x7f9214100600 2024-02-21T23:44:12.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.741+0000 7f91fbfff700 1 --2- 172.21.15.90:0/472268007 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f91fc0a11a0 0x7f91fc0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:44:12.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.741+0000 7f9212ffd700 1 --2- 172.21.15.90:0/472268007 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f91fc0a11a0 0x7f91fc0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:44:12.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.742+0000 7f91fbfff700 1 -- 172.21.15.90:0/472268007 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4566+0+0 (secure 0 0 0) 0x7f92040cb9d0 con 0x7f9214100600 2024-02-21T23:44:12.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.742+0000 7f9212ffd700 1 --2- 172.21.15.90:0/472268007 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f91fc0a11a0 0x7f91fc0a3660 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9208003cf0 tx=0x7f9208005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:44:12.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.745+0000 7f91fbfff700 1 -- 172.21.15.90:0/472268007 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f9204095fe0 con 0x7f9214100600 2024-02-21T23:44:12.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:12 smithi176 ceph-mon[47172]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-02-21T23:44:12.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:12 smithi176 ceph-mon[47172]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-02-21T23:44:12.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:12 smithi176 ceph-mon[47172]: osdmap e36: 6 total, 5 up, 6 in 2024-02-21T23:44:12.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:12 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-21T23:44:12.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:12 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-21T23:44:12.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:12 smithi176 ceph-mon[47172]: osd.5 [v2:172.21.15.178:6808/286274518,v1:172.21.15.178:6809/286274518] boot 2024-02-21T23:44:12.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:12 smithi176 ceph-mon[47172]: osdmap e37: 6 total, 6 up, 6 in 2024-02-21T23:44:12.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:12 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-21T23:44:12.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:12 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:12.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:12 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:12.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:12 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:44:12.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:12 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:44:12.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:12 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:12.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:12 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:12.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:12 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:44:12.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:12 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:12.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:12.948+0000 7f92197d1700 1 -- 172.21.15.90:0/472268007 --> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f92141072b0 con 0x7f91fc0a11a0 2024-02-21T23:44:13.600 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:13 smithi090 ceph-mon[43385]: purged_snaps scrub starts 2024-02-21T23:44:13.600 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:13 smithi090 ceph-mon[43385]: purged_snaps scrub ok 2024-02-21T23:44:13.600 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:13 smithi090 ceph-mon[43385]: Detected new or changed devices on smithi178 2024-02-21T23:44:13.600 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:13 smithi090 ceph-mon[43385]: Adjusting osd_memory_target on smithi178 to 8556M 2024-02-21T23:44:13.601 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:13 smithi090 ceph-mon[43385]: osdmap e38: 6 total, 6 up, 6 in 2024-02-21T23:44:13.601 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:13 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-21T23:44:13.601 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:13 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-21T23:44:13.601 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:13 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:13.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:13 smithi178 ceph-mon[47547]: purged_snaps scrub starts 2024-02-21T23:44:13.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:13 smithi178 ceph-mon[47547]: purged_snaps scrub ok 2024-02-21T23:44:13.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:13 smithi178 ceph-mon[47547]: Detected new or changed devices on smithi178 2024-02-21T23:44:13.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:13 smithi178 ceph-mon[47547]: Adjusting osd_memory_target on smithi178 to 8556M 2024-02-21T23:44:13.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:13 smithi178 ceph-mon[47547]: osdmap e38: 6 total, 6 up, 6 in 2024-02-21T23:44:13.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:13 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-21T23:44:13.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:13 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-21T23:44:13.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:13 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:13.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:13 smithi176 ceph-mon[47172]: purged_snaps scrub starts 2024-02-21T23:44:13.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:13 smithi176 ceph-mon[47172]: purged_snaps scrub ok 2024-02-21T23:44:13.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:13 smithi176 ceph-mon[47172]: Detected new or changed devices on smithi178 2024-02-21T23:44:13.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:13 smithi176 ceph-mon[47172]: Adjusting osd_memory_target on smithi178 to 8556M 2024-02-21T23:44:13.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:13 smithi176 ceph-mon[47172]: osdmap e38: 6 total, 6 up, 6 in 2024-02-21T23:44:13.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:13 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-21T23:44:13.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:13 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-21T23:44:13.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:13 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:14.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:14 smithi178 ceph-mon[47547]: from='client.14367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:44:14.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:14 smithi178 ceph-mon[47547]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2024-02-21T23:44:14.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:14 smithi178 ceph-mon[47547]: osdmap e39: 6 total, 6 up, 6 in 2024-02-21T23:44:14.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:14 smithi090 ceph-mon[43385]: from='client.14367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:44:14.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:14 smithi090 ceph-mon[43385]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2024-02-21T23:44:14.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:14 smithi090 ceph-mon[43385]: osdmap e39: 6 total, 6 up, 6 in 2024-02-21T23:44:14.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:14 smithi176 ceph-mon[47172]: from='client.14367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:44:14.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:14 smithi176 ceph-mon[47172]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2024-02-21T23:44:14.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:14 smithi176 ceph-mon[47172]: osdmap e39: 6 total, 6 up, 6 in 2024-02-21T23:44:15.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:15 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/4070138024' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09bb5393-2efd-4a63-b0e8-99bef5bced13"}]: dispatch 2024-02-21T23:44:15.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:15 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/4070138024' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09bb5393-2efd-4a63-b0e8-99bef5bced13"}]': finished 2024-02-21T23:44:15.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:15 smithi178 ceph-mon[47547]: osdmap e40: 7 total, 6 up, 7 in 2024-02-21T23:44:15.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:15 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-21T23:44:15.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:15 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/466908797' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-21T23:44:15.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:15 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/4070138024' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09bb5393-2efd-4a63-b0e8-99bef5bced13"}]: dispatch 2024-02-21T23:44:15.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:15 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/4070138024' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09bb5393-2efd-4a63-b0e8-99bef5bced13"}]': finished 2024-02-21T23:44:15.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:15 smithi090 ceph-mon[43385]: osdmap e40: 7 total, 6 up, 7 in 2024-02-21T23:44:15.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:15 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-21T23:44:15.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:15 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/466908797' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-21T23:44:15.833 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:15 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/4070138024' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09bb5393-2efd-4a63-b0e8-99bef5bced13"}]: dispatch 2024-02-21T23:44:15.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:15 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/4070138024' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09bb5393-2efd-4a63-b0e8-99bef5bced13"}]': finished 2024-02-21T23:44:15.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:15 smithi176 ceph-mon[47172]: osdmap e40: 7 total, 6 up, 7 in 2024-02-21T23:44:15.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:15 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-21T23:44:15.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:15 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/466908797' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-21T23:44:16.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:16 smithi178 ceph-mon[47547]: pgmap v111: 1 pgs: 1 remapped+peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2024-02-21T23:44:16.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:16 smithi178 ceph-mon[47547]: osdmap e41: 7 total, 6 up, 7 in 2024-02-21T23:44:16.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:16 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-21T23:44:16.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:16 smithi090 ceph-mon[43385]: pgmap v111: 1 pgs: 1 remapped+peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2024-02-21T23:44:16.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:16 smithi090 ceph-mon[43385]: osdmap e41: 7 total, 6 up, 7 in 2024-02-21T23:44:16.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:16 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-21T23:44:16.833 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:16 smithi176 ceph-mon[47172]: pgmap v111: 1 pgs: 1 remapped+peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2024-02-21T23:44:16.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:16 smithi176 ceph-mon[47172]: osdmap e41: 7 total, 6 up, 7 in 2024-02-21T23:44:16.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:16 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-21T23:44:17.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:17 smithi090 ceph-mon[43385]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-02-21T23:44:18.083 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:17 smithi176 ceph-mon[47172]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-02-21T23:44:18.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:17 smithi178 ceph-mon[47547]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-02-21T23:44:20.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:20 smithi090 ceph-mon[43385]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-21T23:44:20.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:20 smithi176 ceph-mon[47172]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-21T23:44:20.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:20 smithi178 ceph-mon[47547]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-21T23:44:21.495 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:21 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-21T23:44:21.496 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:21 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:21.496 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:21 smithi090 ceph-mon[43385]: Deploying daemon osd.6 on smithi090 2024-02-21T23:44:21.833 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:21 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-21T23:44:21.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:21 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:21.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:21 smithi176 ceph-mon[47172]: Deploying daemon osd.6 on smithi090 2024-02-21T23:44:21.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:21 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-21T23:44:21.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:21 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:21.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:21 smithi178 ceph-mon[47547]: Deploying daemon osd.6 on smithi090 2024-02-21T23:44:22.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:22 smithi090 ceph-mon[43385]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-02-21T23:44:22.833 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:22 smithi176 ceph-mon[47172]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-02-21T23:44:22.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:22 smithi178 ceph-mon[47547]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-02-21T23:44:24.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:24 smithi090 ceph-mon[43385]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-02-21T23:44:24.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:24 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:24.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:24 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:24.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:24 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:44:24.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:24 smithi176 ceph-mon[47172]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-02-21T23:44:24.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:24 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:24.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:24 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:24.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:24 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:44:24.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:24 smithi178 ceph-mon[47547]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-02-21T23:44:24.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:24 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:24.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:24 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:24.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:24 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:44:26.725 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:26 smithi090 ceph-mon[43385]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-21T23:44:26.725 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:26 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:26.725 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:26 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:26.725 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:26 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:26.725 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:26 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:44:26.725 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:26 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:26.833 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:26 smithi176 ceph-mon[47172]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-21T23:44:26.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:26 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:26.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:26 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:26.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:26 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:26.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:26 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:44:26.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:26 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:26.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:26 smithi178 ceph-mon[47547]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-21T23:44:26.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:26 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:26.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:26 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:26.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:26 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:26.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:26 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:44:26.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:26 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:27.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:27.045+0000 7f91fbfff700 1 -- 172.21.15.90:0/472268007 <== mgr.14150 v2:172.21.15.90:6800/3697394685 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f92141072b0 con 0x7f91fc0a11a0 2024-02-21T23:44:27.030 INFO:teuthology.orchestra.run.smithi090.stdout:Created osd(s) 6 on host 'smithi090' 2024-02-21T23:44:27.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:27.047+0000 7f92197d1700 1 -- 172.21.15.90:0/472268007 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f91fc0a11a0 msgr2=0x7f91fc0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:44:27.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:27.047+0000 7f92197d1700 1 --2- 172.21.15.90:0/472268007 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f91fc0a11a0 0x7f91fc0a3660 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9208003cf0 tx=0x7f9208005fb0 comp rx=0 tx=0).stop 2024-02-21T23:44:27.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:27.047+0000 7f92197d1700 1 -- 172.21.15.90:0/472268007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9214100600 msgr2=0x7f92141019e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:44:27.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:27.047+0000 7f92197d1700 1 --2- 172.21.15.90:0/472268007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9214100600 0x7f92141019e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f920400c920 tx=0x7f920400cce0 comp rx=0 tx=0).stop 2024-02-21T23:44:27.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:27.047+0000 7f92197d1700 1 -- 172.21.15.90:0/472268007 shutdown_connections 2024-02-21T23:44:27.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:27.047+0000 7f92197d1700 1 --2- 172.21.15.90:0/472268007 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f91fc0a11a0 0x7f91fc0a3660 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:44:27.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:27.047+0000 7f92197d1700 1 --2- 172.21.15.90:0/472268007 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9214100fa0 0x7f9214101f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:44:27.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:27.047+0000 7f92197d1700 1 --2- 172.21.15.90:0/472268007 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f92140ff400 0x7f9214103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:44:27.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:27.047+0000 7f92197d1700 1 --2- 172.21.15.90:0/472268007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9214100600 0x7f92141019e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:44:27.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:27.047+0000 7f92197d1700 1 -- 172.21.15.90:0/472268007 >> 172.21.15.90:0/472268007 conn(0x7f92140fa9c0 msgr2=0x7f9214105b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:44:27.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:27.048+0000 7f92197d1700 1 -- 172.21.15.90:0/472268007 shutdown_connections 2024-02-21T23:44:27.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:44:27.048+0000 7f92197d1700 1 -- 172.21.15.90:0/472268007 wait complete. 2024-02-21T23:44:27.413 DEBUG:teuthology.orchestra.run.smithi090:osd.6> sudo journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@osd.6.service 2024-02-21T23:44:27.415 INFO:tasks.cephadm:Deploying osd.7 on smithi176 with /dev/vg_nvme/lv_2... 2024-02-21T23:44:27.415 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-21T23:44:27.445 INFO:journalctl@ceph.osd.6.smithi090.stdout:-- Logs begin at Wed 2024-02-21 23:24:49 UTC. -- 2024-02-21T23:44:27.646 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.b/config 2024-02-21T23:44:28.278 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:28 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:28.278 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:28 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:44:28.278 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:28 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:28.278 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:28 smithi090 ceph-mon[43385]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-02-21T23:44:28.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:28 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:28.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:28 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:44:28.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:28 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:28.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:28 smithi176 ceph-mon[47172]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-02-21T23:44:28.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:28 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:28.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:28 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:44:28.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:28 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:28.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:28 smithi178 ceph-mon[47547]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-02-21T23:44:28.727 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:44:28 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:44:28.292+0000 7fab5a510640 -1 osd.6 0 log_to_monitors true 2024-02-21T23:44:28.885 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-21T23:44:28.885 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-21T23:44:28.885 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10+0 records in 2024-02-21T23:44:28.885 INFO:teuthology.orchestra.run.smithi176.stderr:10+0 records out 2024-02-21T23:44:28.885 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113063 s, 927 MB/s 2024-02-21T23:44:28.885 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping successful for: 2024-02-21T23:44:29.182 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:29 smithi176 ceph-mon[47172]: from='osd.6 [v2:172.21.15.90:6818/67640620,v1:172.21.15.90:6819/67640620]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-21T23:44:29.265 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:29 smithi090 ceph-mon[43385]: from='osd.6 [v2:172.21.15.90:6818/67640620,v1:172.21.15.90:6819/67640620]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-21T23:44:29.267 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph orch daemon add osd smithi176:vg_nvme/lv_2 2024-02-21T23:44:29.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:29 smithi178 ceph-mon[47547]: from='osd.6 [v2:172.21.15.90:6818/67640620,v1:172.21.15.90:6819/67640620]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-21T23:44:29.486 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.b/config 2024-02-21T23:44:30.247 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:30 smithi176 ceph-mon[47172]: from='osd.6 [v2:172.21.15.90:6818/67640620,v1:172.21.15.90:6819/67640620]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-21T23:44:30.247 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:30 smithi176 ceph-mon[47172]: osdmap e42: 7 total, 6 up, 7 in 2024-02-21T23:44:30.247 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:30 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-21T23:44:30.247 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:30 smithi176 ceph-mon[47172]: from='osd.6 [v2:172.21.15.90:6818/67640620,v1:172.21.15.90:6819/67640620]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-21T23:44:30.247 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:30 smithi176 ceph-mon[47172]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-02-21T23:44:30.247 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:30 smithi176 ceph-mon[47172]: Detected new or changed devices on smithi090 2024-02-21T23:44:30.248 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:30 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:30.248 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:30 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:30.248 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:30 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:44:30.248 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:30 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:44:30.248 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:30 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:44:30.248 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:30 smithi176 ceph-mon[47172]: Adjusting osd_memory_target on smithi090 to 5704M 2024-02-21T23:44:30.248 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:30 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:30.248 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:30 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:30.248 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:30 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:44:30.248 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:30 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:30.415 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.431+0000 7f1658816700 1 -- 172.21.15.176:0/3161992647 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16540ff790 msgr2=0x7f1654109360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:44:30.415 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.431+0000 7f1658816700 1 --2- 172.21.15.176:0/3161992647 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16540ff790 0x7f1654109360 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f164c00b0a0 tx=0x7f164c00b3b0 comp rx=0 tx=0).stop 2024-02-21T23:44:30.415 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.431+0000 7f1658816700 1 -- 172.21.15.176:0/3161992647 shutdown_connections 2024-02-21T23:44:30.415 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.431+0000 7f1658816700 1 --2- 172.21.15.176:0/3161992647 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16540ff790 0x7f1654109360 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:44:30.415 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.431+0000 7f1658816700 1 --2- 172.21.15.176:0/3161992647 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1654102370 0x7f1654102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:44:30.416 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.431+0000 7f1658816700 1 --2- 172.21.15.176:0/3161992647 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f16540fedf0 0x7f16540ff250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:44:30.416 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.431+0000 7f1658816700 1 -- 172.21.15.176:0/3161992647 >> 172.21.15.176:0/3161992647 conn(0x7f16540fa9b0 msgr2=0x7f16540fcdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:44:30.416 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.431+0000 7f1658816700 1 -- 172.21.15.176:0/3161992647 shutdown_connections 2024-02-21T23:44:30.416 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.431+0000 7f1658816700 1 -- 172.21.15.176:0/3161992647 wait complete. 2024-02-21T23:44:30.418 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.432+0000 7f1658816700 1 Processor -- start 2024-02-21T23:44:30.418 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.432+0000 7f1658816700 1 -- start start 2024-02-21T23:44:30.418 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.432+0000 7f1658816700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f16540fedf0 0x7f1654196e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:44:30.418 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.432+0000 7f1658816700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16540ff790 0x7f1654197370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:44:30.418 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.432+0000 7f1658816700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1654102370 0x7f165419e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:44:30.418 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.432+0000 7f1658816700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f165419e8e0 con 0x7f16540fedf0 2024-02-21T23:44:30.418 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.432+0000 7f1658816700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f165419ea20 con 0x7f16540ff790 2024-02-21T23:44:30.418 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.432+0000 7f1658816700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f165419ebe0 con 0x7f1654102370 2024-02-21T23:44:30.418 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.432+0000 7f1651d9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16540ff790 0x7f1654197370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:44:30.419 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.432+0000 7f1651d9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16540ff790 0x7f1654197370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:36878/0 (socket says 172.21.15.176:36878) 2024-02-21T23:44:30.419 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.432+0000 7f1651d9b700 1 -- 172.21.15.176:0/1653387159 learned_addr learned my addr 172.21.15.176:0/1653387159 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-21T23:44:30.419 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.432+0000 7f165259c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f16540fedf0 0x7f1654196e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:44:30.419 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.432+0000 7f1652d9d700 1 --2- 172.21.15.176:0/1653387159 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1654102370 0x7f165419e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:44:30.419 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.432+0000 7f1651d9b700 1 -- 172.21.15.176:0/1653387159 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1654102370 msgr2=0x7f165419e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:44:30.419 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.432+0000 7f1651d9b700 1 --2- 172.21.15.176:0/1653387159 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1654102370 0x7f165419e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:44:30.419 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.432+0000 7f1651d9b700 1 -- 172.21.15.176:0/1653387159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f16540fedf0 msgr2=0x7f1654196e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:44:30.419 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.432+0000 7f1651d9b700 1 --2- 172.21.15.176:0/1653387159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f16540fedf0 0x7f1654196e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:44:30.419 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.432+0000 7f1651d9b700 1 -- 172.21.15.176:0/1653387159 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f164c009d00 con 0x7f16540ff790 2024-02-21T23:44:30.419 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.433+0000 7f1651d9b700 1 --2- 172.21.15.176:0/1653387159 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16540ff790 0x7f1654197370 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f164800b530 tx=0x7f164800b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:44:30.419 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.433+0000 7f16437fe700 1 -- 172.21.15.176:0/1653387159 <== mon.2 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1648011830 con 0x7f16540ff790 2024-02-21T23:44:30.419 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.433+0000 7f1658816700 1 -- 172.21.15.176:0/1653387159 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f165419eec0 con 0x7f16540ff790 2024-02-21T23:44:30.419 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.433+0000 7f16437fe700 1 -- 172.21.15.176:0/1653387159 <== mon.2 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1648011e50 con 0x7f16540ff790 2024-02-21T23:44:30.419 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.433+0000 7f16437fe700 1 -- 172.21.15.176:0/1653387159 <== mon.2 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164800f2e0 con 0x7f16540ff790 2024-02-21T23:44:30.420 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.433+0000 7f1658816700 1 -- 172.21.15.176:0/1653387159 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f165419f330 con 0x7f16540ff790 2024-02-21T23:44:30.420 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.434+0000 7f1658816700 1 -- 172.21.15.176:0/1653387159 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1654108350 con 0x7f16540ff790 2024-02-21T23:44:30.420 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.435+0000 7f16437fe700 1 -- 172.21.15.176:0/1653387159 <== mon.2 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7f1648010380 con 0x7f16540ff790 2024-02-21T23:44:30.421 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.435+0000 7f16437fe700 1 --2- 172.21.15.176:0/1653387159 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f163c0a1120 0x7f163c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:44:30.421 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.436+0000 7f16437fe700 1 -- 172.21.15.176:0/1653387159 <== mon.2 v2:172.21.15.176:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4685+0+0 (secure 0 0 0) 0x7f16480cb6e0 con 0x7f16540ff790 2024-02-21T23:44:30.421 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.436+0000 7f165259c700 1 --2- 172.21.15.176:0/1653387159 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f163c0a1120 0x7f163c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:44:30.421 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.436+0000 7f165259c700 1 --2- 172.21.15.176:0/1653387159 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f163c0a1120 0x7f163c0a35e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1654198270 tx=0x7f1644009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:44:30.424 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.440+0000 7f16437fe700 1 -- 172.21.15.176:0/1653387159 <== mon.2 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f1648095c00 con 0x7f16540ff790 2024-02-21T23:44:30.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:30 smithi178 ceph-mon[47547]: from='osd.6 [v2:172.21.15.90:6818/67640620,v1:172.21.15.90:6819/67640620]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-21T23:44:30.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:30 smithi178 ceph-mon[47547]: osdmap e42: 7 total, 6 up, 7 in 2024-02-21T23:44:30.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:30 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-21T23:44:30.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:30 smithi178 ceph-mon[47547]: from='osd.6 [v2:172.21.15.90:6818/67640620,v1:172.21.15.90:6819/67640620]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-21T23:44:30.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:30 smithi178 ceph-mon[47547]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-02-21T23:44:30.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:30 smithi178 ceph-mon[47547]: Detected new or changed devices on smithi090 2024-02-21T23:44:30.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:30 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:30.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:30 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:30.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:30 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:44:30.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:30 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:44:30.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:30 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:44:30.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:30 smithi178 ceph-mon[47547]: Adjusting osd_memory_target on smithi090 to 5704M 2024-02-21T23:44:30.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:30 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:30.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:30 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:30.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:30 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:44:30.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:30 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:30.477 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:44:30 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:44:30.041+0000 7fab50375700 -1 osd.6 0 waiting for initial osdmap 2024-02-21T23:44:30.477 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:44:30 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:44:30.050+0000 7fab48162700 -1 osd.6 43 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-21T23:44:30.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:30 smithi090 ceph-mon[43385]: from='osd.6 [v2:172.21.15.90:6818/67640620,v1:172.21.15.90:6819/67640620]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-21T23:44:30.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:30 smithi090 ceph-mon[43385]: osdmap e42: 7 total, 6 up, 7 in 2024-02-21T23:44:30.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:30 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-21T23:44:30.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:30 smithi090 ceph-mon[43385]: from='osd.6 [v2:172.21.15.90:6818/67640620,v1:172.21.15.90:6819/67640620]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-21T23:44:30.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:30 smithi090 ceph-mon[43385]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-02-21T23:44:30.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:30 smithi090 ceph-mon[43385]: Detected new or changed devices on smithi090 2024-02-21T23:44:30.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:30 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:30.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:30 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:30.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:30 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:44:30.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:30 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:44:30.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:30 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:44:30.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:30 smithi090 ceph-mon[43385]: Adjusting osd_memory_target on smithi090 to 5704M 2024-02-21T23:44:30.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:30 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:30.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:30 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:30.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:30 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:44:30.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:30 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:30.622 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:30.637+0000 7f1658816700 1 -- 172.21.15.176:0/1653387159 --> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f16540611b0 con 0x7f163c0a1120 2024-02-21T23:44:31.320 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:31 smithi176 ceph-mon[47172]: from='osd.6 [v2:172.21.15.90:6818/67640620,v1:172.21.15.90:6819/67640620]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-02-21T23:44:31.320 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:31 smithi176 ceph-mon[47172]: osdmap e43: 7 total, 6 up, 7 in 2024-02-21T23:44:31.320 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:31 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-21T23:44:31.320 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:31 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-21T23:44:31.320 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:31 smithi176 ceph-mon[47172]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:44:31.321 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:31 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-21T23:44:31.321 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:31 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-21T23:44:31.321 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:31 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:31.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:31 smithi178 ceph-mon[47547]: from='osd.6 [v2:172.21.15.90:6818/67640620,v1:172.21.15.90:6819/67640620]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-02-21T23:44:31.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:31 smithi178 ceph-mon[47547]: osdmap e43: 7 total, 6 up, 7 in 2024-02-21T23:44:31.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:31 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-21T23:44:31.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:31 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-21T23:44:31.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:31 smithi178 ceph-mon[47547]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:44:31.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:31 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-21T23:44:31.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:31 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-21T23:44:31.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:31 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:31.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:31 smithi090 ceph-mon[43385]: from='osd.6 [v2:172.21.15.90:6818/67640620,v1:172.21.15.90:6819/67640620]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-02-21T23:44:31.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:31 smithi090 ceph-mon[43385]: osdmap e43: 7 total, 6 up, 7 in 2024-02-21T23:44:31.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:31 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-21T23:44:31.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:31 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-21T23:44:31.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:31 smithi090 ceph-mon[43385]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:44:31.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:31 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-21T23:44:31.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:31 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-21T23:44:31.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:31 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:32.333 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:32 smithi176 ceph-mon[47172]: purged_snaps scrub starts 2024-02-21T23:44:32.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:32 smithi176 ceph-mon[47172]: purged_snaps scrub ok 2024-02-21T23:44:32.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:32 smithi176 ceph-mon[47172]: osd.6 [v2:172.21.15.90:6818/67640620,v1:172.21.15.90:6819/67640620] boot 2024-02-21T23:44:32.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:32 smithi176 ceph-mon[47172]: osdmap e44: 7 total, 7 up, 7 in 2024-02-21T23:44:32.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:32 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-21T23:44:32.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:32 smithi176 ceph-mon[47172]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 626 GiB / 626 GiB avail 2024-02-21T23:44:32.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:32 smithi178 ceph-mon[47547]: purged_snaps scrub starts 2024-02-21T23:44:32.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:32 smithi178 ceph-mon[47547]: purged_snaps scrub ok 2024-02-21T23:44:32.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:32 smithi178 ceph-mon[47547]: osd.6 [v2:172.21.15.90:6818/67640620,v1:172.21.15.90:6819/67640620] boot 2024-02-21T23:44:32.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:32 smithi178 ceph-mon[47547]: osdmap e44: 7 total, 7 up, 7 in 2024-02-21T23:44:32.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:32 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-21T23:44:32.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:32 smithi178 ceph-mon[47547]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 626 GiB / 626 GiB avail 2024-02-21T23:44:32.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:32 smithi090 ceph-mon[43385]: purged_snaps scrub starts 2024-02-21T23:44:32.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:32 smithi090 ceph-mon[43385]: purged_snaps scrub ok 2024-02-21T23:44:32.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:32 smithi090 ceph-mon[43385]: osd.6 [v2:172.21.15.90:6818/67640620,v1:172.21.15.90:6819/67640620] boot 2024-02-21T23:44:32.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:32 smithi090 ceph-mon[43385]: osdmap e44: 7 total, 7 up, 7 in 2024-02-21T23:44:32.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:32 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-21T23:44:32.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:32 smithi090 ceph-mon[43385]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 626 GiB / 626 GiB avail 2024-02-21T23:44:33.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:33 smithi176 ceph-mon[47172]: osdmap e45: 7 total, 7 up, 7 in 2024-02-21T23:44:33.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:33 smithi176 ceph-mon[47172]: from='client.? 172.21.15.176:0/2150147756' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a859e69-6b63-467b-a3b3-99247d66f5c5"}]: dispatch 2024-02-21T23:44:33.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:33 smithi176 ceph-mon[47172]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a859e69-6b63-467b-a3b3-99247d66f5c5"}]: dispatch 2024-02-21T23:44:33.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:33 smithi176 ceph-mon[47172]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a859e69-6b63-467b-a3b3-99247d66f5c5"}]': finished 2024-02-21T23:44:33.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:33 smithi176 ceph-mon[47172]: osdmap e46: 8 total, 7 up, 8 in 2024-02-21T23:44:33.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:33 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-21T23:44:33.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:33 smithi178 ceph-mon[47547]: osdmap e45: 7 total, 7 up, 7 in 2024-02-21T23:44:33.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:33 smithi178 ceph-mon[47547]: from='client.? 172.21.15.176:0/2150147756' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a859e69-6b63-467b-a3b3-99247d66f5c5"}]: dispatch 2024-02-21T23:44:33.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:33 smithi178 ceph-mon[47547]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a859e69-6b63-467b-a3b3-99247d66f5c5"}]: dispatch 2024-02-21T23:44:33.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:33 smithi178 ceph-mon[47547]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a859e69-6b63-467b-a3b3-99247d66f5c5"}]': finished 2024-02-21T23:44:33.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:33 smithi178 ceph-mon[47547]: osdmap e46: 8 total, 7 up, 8 in 2024-02-21T23:44:33.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:33 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-21T23:44:33.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:33 smithi090 ceph-mon[43385]: osdmap e45: 7 total, 7 up, 7 in 2024-02-21T23:44:33.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:33 smithi090 ceph-mon[43385]: from='client.? 172.21.15.176:0/2150147756' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a859e69-6b63-467b-a3b3-99247d66f5c5"}]: dispatch 2024-02-21T23:44:33.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:33 smithi090 ceph-mon[43385]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a859e69-6b63-467b-a3b3-99247d66f5c5"}]: dispatch 2024-02-21T23:44:33.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:33 smithi090 ceph-mon[43385]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a859e69-6b63-467b-a3b3-99247d66f5c5"}]': finished 2024-02-21T23:44:33.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:33 smithi090 ceph-mon[43385]: osdmap e46: 8 total, 7 up, 8 in 2024-02-21T23:44:33.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:33 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-21T23:44:34.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:34 smithi176 ceph-mon[47172]: from='client.? 172.21.15.176:0/3708770944' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-21T23:44:34.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:34 smithi176 ceph-mon[47172]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-02-21T23:44:34.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:34 smithi178 ceph-mon[47547]: from='client.? 172.21.15.176:0/3708770944' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-21T23:44:34.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:34 smithi178 ceph-mon[47547]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-02-21T23:44:34.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:34 smithi090 ceph-mon[43385]: from='client.? 172.21.15.176:0/3708770944' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-21T23:44:34.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:34 smithi090 ceph-mon[43385]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-02-21T23:44:36.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:36 smithi090 ceph-mon[43385]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-02-21T23:44:36.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:36 smithi176 ceph-mon[47172]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-02-21T23:44:36.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:36 smithi178 ceph-mon[47547]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-02-21T23:44:38.028 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:37 smithi176 ceph-mon[47172]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-02-21T23:44:38.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:37 smithi178 ceph-mon[47547]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-02-21T23:44:38.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:37 smithi090 ceph-mon[43385]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-02-21T23:44:38.893 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:38 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-21T23:44:38.894 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:38 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:38.894 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:38 smithi176 ceph-mon[47172]: Deploying daemon osd.7 on smithi176 2024-02-21T23:44:39.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:38 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-21T23:44:39.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:38 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:39.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:38 smithi178 ceph-mon[47547]: Deploying daemon osd.7 on smithi176 2024-02-21T23:44:39.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:38 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-21T23:44:39.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:38 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:39.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:38 smithi090 ceph-mon[43385]: Deploying daemon osd.7 on smithi176 2024-02-21T23:44:39.880 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:39 smithi176 ceph-mon[47172]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-02-21T23:44:40.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:39 smithi178 ceph-mon[47547]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-02-21T23:44:40.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:39 smithi090 ceph-mon[43385]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-02-21T23:44:42.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:42 smithi090 ceph-mon[43385]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-02-21T23:44:42.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:42 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:42.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:42 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:44:42.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:42 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:42.794 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:42 smithi176 ceph-mon[47172]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-02-21T23:44:42.795 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:42 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:42.795 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:42 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:44:42.795 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:42 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:42.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:42 smithi178 ceph-mon[47547]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-02-21T23:44:42.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:42 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:42.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:42 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:44:42.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:42 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:43.722 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:43.738+0000 7f16437fe700 1 -- 172.21.15.176:0/1653387159 <== mgr.14150 v2:172.21.15.90:6800/3697394685 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f16540611b0 con 0x7f163c0a1120 2024-02-21T23:44:43.723 INFO:teuthology.orchestra.run.smithi176.stdout:Created osd(s) 7 on host 'smithi176' 2024-02-21T23:44:43.725 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:43.741+0000 7f1658816700 1 -- 172.21.15.176:0/1653387159 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f163c0a1120 msgr2=0x7f163c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:44:43.725 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:43.741+0000 7f1658816700 1 --2- 172.21.15.176:0/1653387159 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f163c0a1120 0x7f163c0a35e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1654198270 tx=0x7f1644009260 comp rx=0 tx=0).stop 2024-02-21T23:44:43.725 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:43.741+0000 7f1658816700 1 -- 172.21.15.176:0/1653387159 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16540ff790 msgr2=0x7f1654197370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:44:43.725 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:43.741+0000 7f1658816700 1 --2- 172.21.15.176:0/1653387159 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16540ff790 0x7f1654197370 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f164800b530 tx=0x7f164800b840 comp rx=0 tx=0).stop 2024-02-21T23:44:43.725 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:43.741+0000 7f1658816700 1 -- 172.21.15.176:0/1653387159 shutdown_connections 2024-02-21T23:44:43.725 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:43.741+0000 7f1658816700 1 --2- 172.21.15.176:0/1653387159 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f163c0a1120 0x7f163c0a35e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:44:43.725 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:43.741+0000 7f1658816700 1 --2- 172.21.15.176:0/1653387159 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1654102370 0x7f165419e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:44:43.725 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:43.741+0000 7f1658816700 1 --2- 172.21.15.176:0/1653387159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f16540fedf0 0x7f1654196e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:44:43.725 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:43.741+0000 7f1658816700 1 --2- 172.21.15.176:0/1653387159 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16540ff790 0x7f1654197370 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:44:43.725 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:43.741+0000 7f1658816700 1 -- 172.21.15.176:0/1653387159 >> 172.21.15.176:0/1653387159 conn(0x7f16540fa9b0 msgr2=0x7f1654105490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:44:43.725 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:43.741+0000 7f1658816700 1 -- 172.21.15.176:0/1653387159 shutdown_connections 2024-02-21T23:44:43.725 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:44:43.741+0000 7f1658816700 1 -- 172.21.15.176:0/1653387159 wait complete. 2024-02-21T23:44:44.190 DEBUG:teuthology.orchestra.run.smithi176:osd.7> sudo journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@osd.7.service 2024-02-21T23:44:44.192 INFO:tasks.cephadm:Deploying osd.8 on smithi178 with /dev/vg_nvme/lv_2... 2024-02-21T23:44:44.192 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-21T23:44:44.224 INFO:journalctl@ceph.osd.7.smithi176.stdout:-- Logs begin at Wed 2024-02-21 23:23:58 UTC. -- 2024-02-21T23:44:44.266 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:44 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:44.266 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:44 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:44.266 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:44 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:44.266 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:44 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:44:44.266 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:44 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:44.266 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:44 smithi176 ceph-mon[47172]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-02-21T23:44:44.266 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:44 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:44.267 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:44 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:44.267 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:44 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:44:44.426 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.c/config 2024-02-21T23:44:44.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:44 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:44.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:44 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:44.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:44 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:44.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:44 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:44:44.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:44 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:44.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:44 smithi090 ceph-mon[43385]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-02-21T23:44:44.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:44 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:44.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:44 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:44.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:44 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:44:44.546 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:44 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:44.546 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:44 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:44.546 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:44 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:44.546 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:44 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:44:44.546 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:44 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:44.546 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:44 smithi178 ceph-mon[47547]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-02-21T23:44:44.546 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:44 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:44.546 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:44 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:44.546 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:44 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:44:45.797 INFO:teuthology.orchestra.run.smithi178.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-21T23:44:45.797 INFO:teuthology.orchestra.run.smithi178.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-21T23:44:45.797 INFO:teuthology.orchestra.run.smithi178.stderr: stderr: 10+0 records in 2024-02-21T23:44:45.797 INFO:teuthology.orchestra.run.smithi178.stderr:10+0 records out 2024-02-21T23:44:45.798 INFO:teuthology.orchestra.run.smithi178.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00819568 s, 1.3 GB/s 2024-02-21T23:44:45.798 INFO:teuthology.orchestra.run.smithi178.stderr:--> Zapping successful for: 2024-02-21T23:44:46.084 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:44:45 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:44:45.772+0000 7ff468ce8640 -1 osd.7 0 log_to_monitors true 2024-02-21T23:44:46.201 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph orch daemon add osd smithi178:vg_nvme/lv_2 2024-02-21T23:44:46.431 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.c/config 2024-02-21T23:44:46.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:46 smithi090 ceph-mon[43385]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-02-21T23:44:46.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:46 smithi090 ceph-mon[43385]: from='osd.7 [v2:172.21.15.176:6816/3998561598,v1:172.21.15.176:6817/3998561598]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-21T23:44:46.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:46 smithi090 ceph-mon[43385]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-21T23:44:46.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:46 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:46.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:46 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:46.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:46 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:44:46.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:46 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:44:46.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:46 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:44:46.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:46 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:46.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:46 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:46.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:46 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:44:46.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:46 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:46.758 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:46 smithi178 ceph-mon[47547]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-02-21T23:44:46.758 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:46 smithi178 ceph-mon[47547]: from='osd.7 [v2:172.21.15.176:6816/3998561598,v1:172.21.15.176:6817/3998561598]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-21T23:44:46.758 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:46 smithi178 ceph-mon[47547]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-21T23:44:46.758 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:46 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:46.758 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:46 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:46.759 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:46 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:44:46.759 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:46 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:44:46.759 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:46 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:44:46.759 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:46 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:46.759 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:46 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:46.759 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:46 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:44:46.759 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:46 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:46.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:46 smithi176 ceph-mon[47172]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-02-21T23:44:46.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:46 smithi176 ceph-mon[47172]: from='osd.7 [v2:172.21.15.176:6816/3998561598,v1:172.21.15.176:6817/3998561598]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-21T23:44:46.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:46 smithi176 ceph-mon[47172]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-21T23:44:46.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:46 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:46.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:46 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:46.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:46 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:44:46.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:46 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:44:46.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:46 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:44:46.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:46 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:46.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:46 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:46.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:46 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:44:46.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:46 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:44:47.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.560+0000 7fd21bdae700 1 -- 172.21.15.178:0/2714722405 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd2141020f0 msgr2=0x7fd214102550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:44:47.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.560+0000 7fd21bdae700 1 --2- 172.21.15.178:0/2714722405 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd2141020f0 0x7fd214102550 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd210009990 tx=0x7fd210009ca0 comp rx=0 tx=0).stop 2024-02-21T23:44:47.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.560+0000 7fd21bdae700 1 -- 172.21.15.178:0/2714722405 shutdown_connections 2024-02-21T23:44:47.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.560+0000 7fd21bdae700 1 --2- 172.21.15.178:0/2714722405 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd214102a90 0x7fd214109340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:44:47.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.560+0000 7fd21bdae700 1 --2- 172.21.15.178:0/2714722405 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2140ff070 0x7fd2140ff490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:44:47.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.560+0000 7fd21bdae700 1 --2- 172.21.15.178:0/2714722405 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd2141020f0 0x7fd214102550 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:44:47.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.560+0000 7fd21bdae700 1 -- 172.21.15.178:0/2714722405 >> 172.21.15.178:0/2714722405 conn(0x7fd2140fa9b0 msgr2=0x7fd2140fcdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:44:47.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.560+0000 7fd21bdae700 1 -- 172.21.15.178:0/2714722405 shutdown_connections 2024-02-21T23:44:47.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.560+0000 7fd21bdae700 1 -- 172.21.15.178:0/2714722405 wait complete. 2024-02-21T23:44:47.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.561+0000 7fd21bdae700 1 Processor -- start 2024-02-21T23:44:47.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.561+0000 7fd21bdae700 1 -- start start 2024-02-21T23:44:47.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.561+0000 7fd21bdae700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2140ff070 0x7fd214101a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:44:47.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.561+0000 7fd21bdae700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2141020f0 0x7fd2141000a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:44:47.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.561+0000 7fd21bdae700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd214102a90 0x7fd2141005e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:44:47.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.561+0000 7fd21bdae700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd214100d80 con 0x7fd2140ff070 2024-02-21T23:44:47.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.561+0000 7fd21bdae700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd214100ef0 con 0x7fd2141020f0 2024-02-21T23:44:47.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.561+0000 7fd21bdae700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2141a0d50 con 0x7fd214102a90 2024-02-21T23:44:47.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.561+0000 7fd21a34b700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd214102a90 0x7fd2141005e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:44:47.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.561+0000 7fd21a34b700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd214102a90 0x7fd2141005e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:57712/0 (socket says 172.21.15.178:57712) 2024-02-21T23:44:47.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.561+0000 7fd21a34b700 1 -- 172.21.15.178:0/2504010279 learned_addr learned my addr 172.21.15.178:0/2504010279 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-21T23:44:47.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.561+0000 7fd219b4a700 1 --2- 172.21.15.178:0/2504010279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2140ff070 0x7fd214101a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:44:47.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.561+0000 7fd219349700 1 --2- 172.21.15.178:0/2504010279 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2141020f0 0x7fd2141000a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:44:47.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.561+0000 7fd21a34b700 1 -- 172.21.15.178:0/2504010279 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2141020f0 msgr2=0x7fd2141000a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:44:47.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.561+0000 7fd21a34b700 1 --2- 172.21.15.178:0/2504010279 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2141020f0 0x7fd2141000a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:44:47.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.561+0000 7fd21a34b700 1 -- 172.21.15.178:0/2504010279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2140ff070 msgr2=0x7fd214101a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:44:47.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.561+0000 7fd21a34b700 1 --2- 172.21.15.178:0/2504010279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2140ff070 0x7fd214101a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:44:47.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.561+0000 7fd21a34b700 1 -- 172.21.15.178:0/2504010279 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2100095f0 con 0x7fd214102a90 2024-02-21T23:44:47.549 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.561+0000 7fd219349700 1 --2- 172.21.15.178:0/2504010279 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2141020f0 0x7fd2141000a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:44:47.549 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.561+0000 7fd21a34b700 1 --2- 172.21.15.178:0/2504010279 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd214102a90 0x7fd2141005e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd20c00ada0 tx=0x7fd20c00c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:44:47.549 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.562+0000 7fd20affd700 1 -- 172.21.15.178:0/2504010279 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd20c00f070 con 0x7fd214102a90 2024-02-21T23:44:47.549 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.562+0000 7fd20affd700 1 -- 172.21.15.178:0/2504010279 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd20c007b30 con 0x7fd214102a90 2024-02-21T23:44:47.549 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.562+0000 7fd21bdae700 1 -- 172.21.15.178:0/2504010279 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2141a0fe0 con 0x7fd214102a90 2024-02-21T23:44:47.549 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.562+0000 7fd20affd700 1 -- 172.21.15.178:0/2504010279 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd20c0053e0 con 0x7fd214102a90 2024-02-21T23:44:47.549 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.562+0000 7fd21bdae700 1 -- 172.21.15.178:0/2504010279 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2141a1450 con 0x7fd214102a90 2024-02-21T23:44:47.549 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.563+0000 7fd21bdae700 1 -- 172.21.15.178:0/2504010279 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2141083d0 con 0x7fd214102a90 2024-02-21T23:44:47.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.564+0000 7fd20affd700 1 -- 172.21.15.178:0/2504010279 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7fd20c00c980 con 0x7fd214102a90 2024-02-21T23:44:47.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.564+0000 7fd20affd700 1 --2- 172.21.15.178:0/2504010279 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fd2000a11a0 0x7fd2000a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:44:47.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.564+0000 7fd20affd700 1 -- 172.21.15.178:0/2504010279 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5117+0+0 (secure 0 0 0) 0x7fd20c0cb790 con 0x7fd214102a90 2024-02-21T23:44:47.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.565+0000 7fd219b4a700 1 --2- 172.21.15.178:0/2504010279 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fd2000a11a0 0x7fd2000a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:44:47.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.565+0000 7fd219b4a700 1 --2- 172.21.15.178:0/2504010279 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fd2000a11a0 0x7fd2000a3660 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd214068420 tx=0x7fd20400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:44:47.551 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.568+0000 7fd20affd700 1 -- 172.21.15.178:0/2504010279 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fd20c095c30 con 0x7fd214102a90 2024-02-21T23:44:47.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:47 smithi090 ceph-mon[43385]: Detected new or changed devices on smithi176 2024-02-21T23:44:47.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:47 smithi090 ceph-mon[43385]: Adjusting osd_memory_target on smithi176 to 5704M 2024-02-21T23:44:47.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:47 smithi090 ceph-mon[43385]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-21T23:44:47.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:47 smithi090 ceph-mon[43385]: osdmap e47: 8 total, 7 up, 8 in 2024-02-21T23:44:47.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:47 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-21T23:44:47.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:47 smithi090 ceph-mon[43385]: from='osd.7 [v2:172.21.15.176:6816/3998561598,v1:172.21.15.176:6817/3998561598]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-21T23:44:47.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:47 smithi090 ceph-mon[43385]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-21T23:44:47.749 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:44:47.766+0000 7fd21bdae700 1 -- 172.21.15.178:0/2504010279 --> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fd2140611b0 con 0x7fd2000a11a0 2024-02-21T23:44:47.833 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:44:47 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:44:47.466+0000 7ff45e34c700 -1 osd.7 0 waiting for initial osdmap 2024-02-21T23:44:47.833 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:44:47 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:44:47.480+0000 7ff45813d700 -1 osd.7 48 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-21T23:44:47.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:47 smithi176 ceph-mon[47172]: Detected new or changed devices on smithi176 2024-02-21T23:44:47.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:47 smithi176 ceph-mon[47172]: Adjusting osd_memory_target on smithi176 to 5704M 2024-02-21T23:44:47.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:47 smithi176 ceph-mon[47172]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-21T23:44:47.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:47 smithi176 ceph-mon[47172]: osdmap e47: 8 total, 7 up, 8 in 2024-02-21T23:44:47.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:47 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-21T23:44:47.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:47 smithi176 ceph-mon[47172]: from='osd.7 [v2:172.21.15.176:6816/3998561598,v1:172.21.15.176:6817/3998561598]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-21T23:44:47.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:47 smithi176 ceph-mon[47172]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-21T23:44:47.923 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:47 smithi178 ceph-mon[47547]: Detected new or changed devices on smithi176 2024-02-21T23:44:47.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:47 smithi178 ceph-mon[47547]: Adjusting osd_memory_target on smithi176 to 5704M 2024-02-21T23:44:47.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:47 smithi178 ceph-mon[47547]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-21T23:44:47.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:47 smithi178 ceph-mon[47547]: osdmap e47: 8 total, 7 up, 8 in 2024-02-21T23:44:47.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:47 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-21T23:44:47.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:47 smithi178 ceph-mon[47547]: from='osd.7 [v2:172.21.15.176:6816/3998561598,v1:172.21.15.176:6817/3998561598]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-21T23:44:47.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:47 smithi178 ceph-mon[47547]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-21T23:44:48.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:48 smithi090 ceph-mon[43385]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-02-21T23:44:48.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:48 smithi090 ceph-mon[43385]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-02-21T23:44:48.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:48 smithi090 ceph-mon[43385]: osdmap e48: 8 total, 7 up, 8 in 2024-02-21T23:44:48.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:48 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-21T23:44:48.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:48 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-21T23:44:48.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:48 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-21T23:44:48.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:48 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-21T23:44:48.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:48 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:48.820 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:48 smithi178 ceph-mon[47547]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-02-21T23:44:48.820 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:48 smithi178 ceph-mon[47547]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-02-21T23:44:48.820 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:48 smithi178 ceph-mon[47547]: osdmap e48: 8 total, 7 up, 8 in 2024-02-21T23:44:48.820 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:48 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-21T23:44:48.821 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:48 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-21T23:44:48.821 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:48 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-21T23:44:48.821 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:48 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-21T23:44:48.821 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:48 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:48.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:48 smithi176 ceph-mon[47172]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-02-21T23:44:48.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:48 smithi176 ceph-mon[47172]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-02-21T23:44:48.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:48 smithi176 ceph-mon[47172]: osdmap e48: 8 total, 7 up, 8 in 2024-02-21T23:44:48.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:48 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-21T23:44:48.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:48 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-21T23:44:48.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:48 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-21T23:44:48.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:48 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-21T23:44:48.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:48 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:49.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:49 smithi090 ceph-mon[43385]: purged_snaps scrub starts 2024-02-21T23:44:49.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:49 smithi090 ceph-mon[43385]: purged_snaps scrub ok 2024-02-21T23:44:49.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:49 smithi090 ceph-mon[43385]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:44:49.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:49 smithi090 ceph-mon[43385]: osd.7 [v2:172.21.15.176:6816/3998561598,v1:172.21.15.176:6817/3998561598] boot 2024-02-21T23:44:49.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:49 smithi090 ceph-mon[43385]: osdmap e49: 8 total, 8 up, 8 in 2024-02-21T23:44:49.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:49 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-21T23:44:49.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:49 smithi176 ceph-mon[47172]: purged_snaps scrub starts 2024-02-21T23:44:49.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:49 smithi176 ceph-mon[47172]: purged_snaps scrub ok 2024-02-21T23:44:49.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:49 smithi176 ceph-mon[47172]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:44:49.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:49 smithi176 ceph-mon[47172]: osd.7 [v2:172.21.15.176:6816/3998561598,v1:172.21.15.176:6817/3998561598] boot 2024-02-21T23:44:49.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:49 smithi176 ceph-mon[47172]: osdmap e49: 8 total, 8 up, 8 in 2024-02-21T23:44:49.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:49 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-21T23:44:49.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:49 smithi178 ceph-mon[47547]: purged_snaps scrub starts 2024-02-21T23:44:49.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:49 smithi178 ceph-mon[47547]: purged_snaps scrub ok 2024-02-21T23:44:49.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:49 smithi178 ceph-mon[47547]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:44:49.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:49 smithi178 ceph-mon[47547]: osd.7 [v2:172.21.15.176:6816/3998561598,v1:172.21.15.176:6817/3998561598] boot 2024-02-21T23:44:49.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:49 smithi178 ceph-mon[47547]: osdmap e49: 8 total, 8 up, 8 in 2024-02-21T23:44:49.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:49 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-21T23:44:50.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:50 smithi090 ceph-mon[43385]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-02-21T23:44:50.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:50 smithi090 ceph-mon[43385]: osdmap e50: 8 total, 8 up, 8 in 2024-02-21T23:44:50.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:50 smithi090 ceph-mon[43385]: from='client.? 172.21.15.178:0/3184414482' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc23144e-22a5-4894-a8b5-9323383574d1"}]: dispatch 2024-02-21T23:44:50.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:50 smithi090 ceph-mon[43385]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc23144e-22a5-4894-a8b5-9323383574d1"}]: dispatch 2024-02-21T23:44:50.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:50 smithi090 ceph-mon[43385]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc23144e-22a5-4894-a8b5-9323383574d1"}]': finished 2024-02-21T23:44:50.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:50 smithi090 ceph-mon[43385]: osdmap e51: 9 total, 8 up, 9 in 2024-02-21T23:44:50.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:50 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-21T23:44:50.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:50 smithi090 ceph-mon[43385]: from='client.? 172.21.15.178:0/25748415' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-21T23:44:50.833 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:50 smithi176 ceph-mon[47172]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-02-21T23:44:50.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:50 smithi176 ceph-mon[47172]: osdmap e50: 8 total, 8 up, 8 in 2024-02-21T23:44:50.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:50 smithi176 ceph-mon[47172]: from='client.? 172.21.15.178:0/3184414482' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc23144e-22a5-4894-a8b5-9323383574d1"}]: dispatch 2024-02-21T23:44:50.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:50 smithi176 ceph-mon[47172]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc23144e-22a5-4894-a8b5-9323383574d1"}]: dispatch 2024-02-21T23:44:50.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:50 smithi176 ceph-mon[47172]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc23144e-22a5-4894-a8b5-9323383574d1"}]': finished 2024-02-21T23:44:50.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:50 smithi176 ceph-mon[47172]: osdmap e51: 9 total, 8 up, 9 in 2024-02-21T23:44:50.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:50 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-21T23:44:50.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:50 smithi176 ceph-mon[47172]: from='client.? 172.21.15.178:0/25748415' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-21T23:44:50.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:50 smithi178 ceph-mon[47547]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-02-21T23:44:50.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:50 smithi178 ceph-mon[47547]: osdmap e50: 8 total, 8 up, 8 in 2024-02-21T23:44:50.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:50 smithi178 ceph-mon[47547]: from='client.? 172.21.15.178:0/3184414482' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc23144e-22a5-4894-a8b5-9323383574d1"}]: dispatch 2024-02-21T23:44:50.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:50 smithi178 ceph-mon[47547]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc23144e-22a5-4894-a8b5-9323383574d1"}]: dispatch 2024-02-21T23:44:50.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:50 smithi178 ceph-mon[47547]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc23144e-22a5-4894-a8b5-9323383574d1"}]': finished 2024-02-21T23:44:50.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:50 smithi178 ceph-mon[47547]: osdmap e51: 9 total, 8 up, 9 in 2024-02-21T23:44:50.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:50 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-21T23:44:50.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:50 smithi178 ceph-mon[47547]: from='client.? 172.21.15.178:0/25748415' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-21T23:44:51.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:51 smithi178 ceph-mon[47547]: osdmap e52: 9 total, 8 up, 9 in 2024-02-21T23:44:51.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:51 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-21T23:44:51.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:51 smithi090 ceph-mon[43385]: osdmap e52: 9 total, 8 up, 9 in 2024-02-21T23:44:51.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:51 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-21T23:44:52.083 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:51 smithi176 ceph-mon[47172]: osdmap e52: 9 total, 8 up, 9 in 2024-02-21T23:44:52.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:51 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-21T23:44:52.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:52 smithi178 ceph-mon[47547]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-02-21T23:44:52.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:52 smithi178 ceph-mon[47547]: osdmap e53: 9 total, 8 up, 9 in 2024-02-21T23:44:52.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:52 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-21T23:44:52.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:52 smithi090 ceph-mon[43385]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-02-21T23:44:52.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:52 smithi090 ceph-mon[43385]: osdmap e53: 9 total, 8 up, 9 in 2024-02-21T23:44:52.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:52 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-21T23:44:53.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:52 smithi176 ceph-mon[47172]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-02-21T23:44:53.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:52 smithi176 ceph-mon[47172]: osdmap e53: 9 total, 8 up, 9 in 2024-02-21T23:44:53.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:52 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-21T23:44:54.842 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:54 smithi178 ceph-mon[47547]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-02-21T23:44:54.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:54 smithi090 ceph-mon[43385]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-02-21T23:44:55.083 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:54 smithi176 ceph-mon[47172]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-02-21T23:44:56.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:56 smithi178 ceph-mon[47547]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-02-21T23:44:56.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:56 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-02-21T23:44:56.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:56 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:56.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:56 smithi178 ceph-mon[47547]: Deploying daemon osd.8 on smithi178 2024-02-21T23:44:56.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:56 smithi090 ceph-mon[43385]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-02-21T23:44:56.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:56 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-02-21T23:44:56.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:56 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:56.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:56 smithi090 ceph-mon[43385]: Deploying daemon osd.8 on smithi178 2024-02-21T23:44:57.083 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:56 smithi176 ceph-mon[47172]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-02-21T23:44:57.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:56 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-02-21T23:44:57.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:56 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:44:57.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:56 smithi176 ceph-mon[47172]: Deploying daemon osd.8 on smithi178 2024-02-21T23:44:57.897 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:44:57 smithi178 ceph-mon[47547]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-02-21T23:44:58.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:44:57 smithi176 ceph-mon[47172]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-02-21T23:44:58.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:44:57 smithi090 ceph-mon[43385]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-02-21T23:45:00.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:00 smithi178 ceph-mon[47547]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-02-21T23:45:00.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:00 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:00.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:00 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:45:00.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:00 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:00.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:00 smithi090 ceph-mon[43385]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-02-21T23:45:00.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:00 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:00.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:00 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:45:00.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:00 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:01.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:00 smithi176 ceph-mon[47172]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-02-21T23:45:01.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:00 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:01.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:00 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:45:01.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:00 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:02.254 INFO:teuthology.orchestra.run.smithi178.stdout:Created osd(s) 8 on host 'smithi178' 2024-02-21T23:45:02.255 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:02.269+0000 7fd20affd700 1 -- 172.21.15.178:0/2504010279 <== mgr.14150 v2:172.21.15.90:6800/3697394685 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd2140611b0 con 0x7fd2000a11a0 2024-02-21T23:45:02.256 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:02.272+0000 7fd21bdae700 1 -- 172.21.15.178:0/2504010279 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fd2000a11a0 msgr2=0x7fd2000a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:45:02.256 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:02.272+0000 7fd21bdae700 1 --2- 172.21.15.178:0/2504010279 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fd2000a11a0 0x7fd2000a3660 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd214068420 tx=0x7fd20400b410 comp rx=0 tx=0).stop 2024-02-21T23:45:02.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:02.272+0000 7fd21bdae700 1 -- 172.21.15.178:0/2504010279 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd214102a90 msgr2=0x7fd2141005e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:45:02.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:02.272+0000 7fd21bdae700 1 --2- 172.21.15.178:0/2504010279 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd214102a90 0x7fd2141005e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd20c00ada0 tx=0x7fd20c00c330 comp rx=0 tx=0).stop 2024-02-21T23:45:02.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:02.272+0000 7fd21bdae700 1 -- 172.21.15.178:0/2504010279 shutdown_connections 2024-02-21T23:45:02.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:02.272+0000 7fd21bdae700 1 --2- 172.21.15.178:0/2504010279 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fd2000a11a0 0x7fd2000a3660 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:02.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:02.272+0000 7fd21bdae700 1 --2- 172.21.15.178:0/2504010279 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd214102a90 0x7fd2141005e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:02.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:02.272+0000 7fd21bdae700 1 --2- 172.21.15.178:0/2504010279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2140ff070 0x7fd214101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:02.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:02.272+0000 7fd21bdae700 1 --2- 172.21.15.178:0/2504010279 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2141020f0 0x7fd2141000a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:02.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:02.272+0000 7fd21bdae700 1 -- 172.21.15.178:0/2504010279 >> 172.21.15.178:0/2504010279 conn(0x7fd2140fa9b0 msgr2=0x7fd214105510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:45:02.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:02.272+0000 7fd21bdae700 1 -- 172.21.15.178:0/2504010279 shutdown_connections 2024-02-21T23:45:02.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:02.272+0000 7fd21bdae700 1 -- 172.21.15.178:0/2504010279 wait complete. 2024-02-21T23:45:02.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:02 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:02.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:02 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:02.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:02 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:02.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:02 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:45:02.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:02 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:02.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:02 smithi178 ceph-mon[47547]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-02-21T23:45:02.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:02 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:02.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:02 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:02.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:02 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:02.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:02 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:45:02.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:02 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:02.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:02 smithi090 ceph-mon[43385]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-02-21T23:45:02.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:02 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:02.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:02 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:02.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:02 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:02.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:02 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:45:02.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:02 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:02.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:02 smithi176 ceph-mon[47172]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-02-21T23:45:02.680 DEBUG:teuthology.orchestra.run.smithi178:osd.8> sudo journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@osd.8.service 2024-02-21T23:45:02.682 INFO:tasks.cephadm:Deploying osd.9 on smithi090 with /dev/vg_nvme/lv_1... 2024-02-21T23:45:02.682 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-21T23:45:02.715 INFO:journalctl@ceph.osd.8.smithi178.stdout:-- Logs begin at Wed 2024-02-21 23:24:26 UTC. -- 2024-02-21T23:45:02.926 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.a/config 2024-02-21T23:45:03.373 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:03 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:03.373 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:03 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:03.374 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:03 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:45:03.583 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:03 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:03.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:03 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:03.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:03 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:45:03.637 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:03 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:03.637 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:03 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:03.637 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:03 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:45:03.924 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:45:03 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:45:03.667+0000 7ff4829bd640 -1 osd.8 0 log_to_monitors true 2024-02-21T23:45:04.365 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-21T23:45:04.365 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-21T23:45:04.365 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10+0 records in 2024-02-21T23:45:04.365 INFO:teuthology.orchestra.run.smithi090.stderr:10+0 records out 2024-02-21T23:45:04.366 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112674 s, 931 MB/s 2024-02-21T23:45:04.366 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping successful for: 2024-02-21T23:45:04.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:04 smithi176 ceph-mon[47172]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-02-21T23:45:04.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:04 smithi176 ceph-mon[47172]: from='osd.8 [v2:172.21.15.178:6816/490229963,v1:172.21.15.178:6817/490229963]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-02-21T23:45:04.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:04 smithi176 ceph-mon[47172]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-02-21T23:45:04.610 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:04 smithi090 ceph-mon[43385]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-02-21T23:45:04.610 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:04 smithi090 ceph-mon[43385]: from='osd.8 [v2:172.21.15.178:6816/490229963,v1:172.21.15.178:6817/490229963]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-02-21T23:45:04.610 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:04 smithi090 ceph-mon[43385]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-02-21T23:45:04.651 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:04 smithi178 ceph-mon[47547]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-02-21T23:45:04.651 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:04 smithi178 ceph-mon[47547]: from='osd.8 [v2:172.21.15.178:6816/490229963,v1:172.21.15.178:6817/490229963]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-02-21T23:45:04.651 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:04 smithi178 ceph-mon[47547]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-02-21T23:45:04.813 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph orch daemon add osd smithi090:vg_nvme/lv_1 2024-02-21T23:45:05.031 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.a/config 2024-02-21T23:45:05.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:05 smithi090 ceph-mon[43385]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-02-21T23:45:05.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:05 smithi090 ceph-mon[43385]: osdmap e54: 9 total, 8 up, 9 in 2024-02-21T23:45:05.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:05 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-21T23:45:05.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:05 smithi090 ceph-mon[43385]: from='osd.8 [v2:172.21.15.178:6816/490229963,v1:172.21.15.178:6817/490229963]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-02-21T23:45:05.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:05 smithi090 ceph-mon[43385]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-02-21T23:45:05.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:05 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:05.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:05 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:05.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:05 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:45:05.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:05 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:45:05.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:05 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:45:05.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:05 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:05.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:05 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:05.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:05 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:45:05.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:05 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:05.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:05 smithi176 ceph-mon[47172]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-02-21T23:45:05.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:05 smithi176 ceph-mon[47172]: osdmap e54: 9 total, 8 up, 9 in 2024-02-21T23:45:05.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:05 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-21T23:45:05.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:05 smithi176 ceph-mon[47172]: from='osd.8 [v2:172.21.15.178:6816/490229963,v1:172.21.15.178:6817/490229963]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-02-21T23:45:05.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:05 smithi176 ceph-mon[47172]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-02-21T23:45:05.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:05 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:05.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:05 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:05.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:05 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:45:05.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:05 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:45:05.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:05 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:45:05.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:05 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:05.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:05 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:05.585 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:05 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:45:05.585 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:05 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:05.674 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:45:05 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:45:05.262+0000 7ff478021700 -1 osd.8 0 waiting for initial osdmap 2024-02-21T23:45:05.674 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:45:05 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:45:05.276+0000 7ff473e16700 -1 osd.8 55 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-21T23:45:05.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:05 smithi178 ceph-mon[47547]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-02-21T23:45:05.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:05 smithi178 ceph-mon[47547]: osdmap e54: 9 total, 8 up, 9 in 2024-02-21T23:45:05.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:05 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-21T23:45:05.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:05 smithi178 ceph-mon[47547]: from='osd.8 [v2:172.21.15.178:6816/490229963,v1:172.21.15.178:6817/490229963]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-02-21T23:45:05.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:05 smithi178 ceph-mon[47547]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-02-21T23:45:05.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:05 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:05.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:05 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:05.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:05 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:45:05.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:05 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:45:05.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:05 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:45:05.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:05 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:05.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:05 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:05.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:05 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:45:05.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:05 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:06.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.044+0000 7fcc7982c700 1 -- 172.21.15.90:0/229615074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc740ffc90 msgr2=0x7fcc741000b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:45:06.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.044+0000 7fcc7982c700 1 --2- 172.21.15.90:0/229615074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc740ffc90 0x7fcc741000b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcc680099c0 tx=0x7fcc68009cd0 comp rx=0 tx=0).stop 2024-02-21T23:45:06.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.044+0000 7fcc7982c700 1 -- 172.21.15.90:0/229615074 shutdown_connections 2024-02-21T23:45:06.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.044+0000 7fcc7982c700 1 --2- 172.21.15.90:0/229615074 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc74101830 0x7fcc741080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:06.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.044+0000 7fcc7982c700 1 --2- 172.21.15.90:0/229615074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc740ffc90 0x7fcc741000b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:06.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.044+0000 7fcc7982c700 1 --2- 172.21.15.90:0/229615074 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcc74100e90 0x7fcc741012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:06.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.044+0000 7fcc7982c700 1 -- 172.21.15.90:0/229615074 >> 172.21.15.90:0/229615074 conn(0x7fcc740fb230 msgr2=0x7fcc740fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:45:06.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.044+0000 7fcc7982c700 1 -- 172.21.15.90:0/229615074 shutdown_connections 2024-02-21T23:45:06.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.044+0000 7fcc7982c700 1 -- 172.21.15.90:0/229615074 wait complete. 2024-02-21T23:45:06.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.045+0000 7fcc7982c700 1 Processor -- start 2024-02-21T23:45:06.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.045+0000 7fcc7982c700 1 -- start start 2024-02-21T23:45:06.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.045+0000 7fcc7982c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc740ffc90 0x7fcc74196de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:45:06.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.045+0000 7fcc7982c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc74100e90 0x7fcc74197320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:45:06.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.045+0000 7fcc7982c700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcc74101830 0x7fcc7419e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:45:06.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.045+0000 7fcc7982c700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7419e8e0 con 0x7fcc74100e90 2024-02-21T23:45:06.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.045+0000 7fcc7982c700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7419ea20 con 0x7fcc740ffc90 2024-02-21T23:45:06.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.045+0000 7fcc7982c700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7419ebe0 con 0x7fcc74101830 2024-02-21T23:45:06.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.045+0000 7fcc727fc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc74100e90 0x7fcc74197320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:45:06.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.045+0000 7fcc727fc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc74100e90 0x7fcc74197320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33198/0 (socket says 172.21.15.90:33198) 2024-02-21T23:45:06.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.045+0000 7fcc727fc700 1 -- 172.21.15.90:0/2561045598 learned_addr learned my addr 172.21.15.90:0/2561045598 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:45:06.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.045+0000 7fcc737fe700 1 --2- 172.21.15.90:0/2561045598 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcc74101830 0x7fcc7419e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:45:06.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.045+0000 7fcc72ffd700 1 --2- 172.21.15.90:0/2561045598 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc740ffc90 0x7fcc74196de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:45:06.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.045+0000 7fcc727fc700 1 -- 172.21.15.90:0/2561045598 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcc74101830 msgr2=0x7fcc7419e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:45:06.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.045+0000 7fcc727fc700 1 --2- 172.21.15.90:0/2561045598 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcc74101830 0x7fcc7419e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:06.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.045+0000 7fcc727fc700 1 -- 172.21.15.90:0/2561045598 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc740ffc90 msgr2=0x7fcc74196de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:45:06.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.045+0000 7fcc727fc700 1 --2- 172.21.15.90:0/2561045598 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc740ffc90 0x7fcc74196de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:06.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.045+0000 7fcc727fc700 1 -- 172.21.15.90:0/2561045598 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc680095f0 con 0x7fcc74100e90 2024-02-21T23:45:06.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.046+0000 7fcc727fc700 1 --2- 172.21.15.90:0/2561045598 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc74100e90 0x7fcc74197320 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fcc6400b530 tx=0x7fcc6400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:45:06.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.046+0000 7fcc7882a700 1 -- 172.21.15.90:0/2561045598 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc64011830 con 0x7fcc74100e90 2024-02-21T23:45:06.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.046+0000 7fcc7982c700 1 -- 172.21.15.90:0/2561045598 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc7419eec0 con 0x7fcc74100e90 2024-02-21T23:45:06.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.046+0000 7fcc7882a700 1 -- 172.21.15.90:0/2561045598 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcc64011e50 con 0x7fcc74100e90 2024-02-21T23:45:06.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.046+0000 7fcc7882a700 1 -- 172.21.15.90:0/2561045598 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6400f2e0 con 0x7fcc74100e90 2024-02-21T23:45:06.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.046+0000 7fcc7982c700 1 -- 172.21.15.90:0/2561045598 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc7419f2d0 con 0x7fcc74100e90 2024-02-21T23:45:06.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.047+0000 7fcc7982c700 1 -- 172.21.15.90:0/2561045598 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc7404ea40 con 0x7fcc74100e90 2024-02-21T23:45:06.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.048+0000 7fcc7882a700 1 -- 172.21.15.90:0/2561045598 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7fcc64010380 con 0x7fcc74100e90 2024-02-21T23:45:06.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.048+0000 7fcc7882a700 1 --2- 172.21.15.90:0/2561045598 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fcc5c0a1120 0x7fcc5c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:45:06.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.048+0000 7fcc7882a700 1 -- 172.21.15.90:0/2561045598 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5549+0+0 (secure 0 0 0) 0x7fcc640cb9b0 con 0x7fcc74100e90 2024-02-21T23:45:06.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.048+0000 7fcc72ffd700 1 --2- 172.21.15.90:0/2561045598 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fcc5c0a1120 0x7fcc5c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:45:06.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.049+0000 7fcc72ffd700 1 --2- 172.21.15.90:0/2561045598 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fcc5c0a1120 0x7fcc5c0a35e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fcc68009990 tx=0x7fcc68005c00 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:45:06.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.053+0000 7fcc7882a700 1 -- 172.21.15.90:0/2561045598 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fcc64095c10 con 0x7fcc74100e90 2024-02-21T23:45:06.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:06.254+0000 7fcc7982c700 1 -- 172.21.15.90:0/2561045598 --> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fcc7419f550 con 0x7fcc5c0a1120 2024-02-21T23:45:06.583 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:06 smithi176 ceph-mon[47172]: Detected new or changed devices on smithi178 2024-02-21T23:45:06.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:06 smithi176 ceph-mon[47172]: Adjusting osd_memory_target on smithi178 to 5704M 2024-02-21T23:45:06.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:06 smithi176 ceph-mon[47172]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-02-21T23:45:06.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:06 smithi176 ceph-mon[47172]: osdmap e55: 9 total, 8 up, 9 in 2024-02-21T23:45:06.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:06 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-21T23:45:06.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:06 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-21T23:45:06.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:06 smithi176 ceph-mon[47172]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-02-21T23:45:06.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:06 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-21T23:45:06.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:06 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-21T23:45:06.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:06 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-21T23:45:06.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:06 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:06.587 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:06 smithi090 ceph-mon[43385]: Detected new or changed devices on smithi178 2024-02-21T23:45:06.587 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:06 smithi090 ceph-mon[43385]: Adjusting osd_memory_target on smithi178 to 5704M 2024-02-21T23:45:06.587 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:06 smithi090 ceph-mon[43385]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-02-21T23:45:06.587 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:06 smithi090 ceph-mon[43385]: osdmap e55: 9 total, 8 up, 9 in 2024-02-21T23:45:06.587 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:06 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-21T23:45:06.588 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:06 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-21T23:45:06.588 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:06 smithi090 ceph-mon[43385]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-02-21T23:45:06.588 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:06 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-21T23:45:06.588 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:06 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-21T23:45:06.588 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:06 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-21T23:45:06.588 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:06 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:06.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:06 smithi178 ceph-mon[47547]: Detected new or changed devices on smithi178 2024-02-21T23:45:06.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:06 smithi178 ceph-mon[47547]: Adjusting osd_memory_target on smithi178 to 5704M 2024-02-21T23:45:06.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:06 smithi178 ceph-mon[47547]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-02-21T23:45:06.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:06 smithi178 ceph-mon[47547]: osdmap e55: 9 total, 8 up, 9 in 2024-02-21T23:45:06.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:06 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-21T23:45:06.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:06 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-21T23:45:06.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:06 smithi178 ceph-mon[47547]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-02-21T23:45:06.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:06 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-21T23:45:06.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:06 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-21T23:45:06.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:06 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-21T23:45:06.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:06 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:07.583 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:07 smithi176 ceph-mon[47172]: purged_snaps scrub starts 2024-02-21T23:45:07.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:07 smithi176 ceph-mon[47172]: purged_snaps scrub ok 2024-02-21T23:45:07.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:07 smithi176 ceph-mon[47172]: from='client.14433 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:45:07.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:07 smithi176 ceph-mon[47172]: osd.8 [v2:172.21.15.178:6816/490229963,v1:172.21.15.178:6817/490229963] boot 2024-02-21T23:45:07.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:07 smithi176 ceph-mon[47172]: osdmap e56: 9 total, 9 up, 9 in 2024-02-21T23:45:07.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:07 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-21T23:45:07.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:07 smithi176 ceph-mon[47172]: osdmap e57: 9 total, 9 up, 9 in 2024-02-21T23:45:07.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:07 smithi178 ceph-mon[47547]: purged_snaps scrub starts 2024-02-21T23:45:07.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:07 smithi178 ceph-mon[47547]: purged_snaps scrub ok 2024-02-21T23:45:07.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:07 smithi178 ceph-mon[47547]: from='client.14433 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:45:07.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:07 smithi178 ceph-mon[47547]: osd.8 [v2:172.21.15.178:6816/490229963,v1:172.21.15.178:6817/490229963] boot 2024-02-21T23:45:07.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:07 smithi178 ceph-mon[47547]: osdmap e56: 9 total, 9 up, 9 in 2024-02-21T23:45:07.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:07 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-21T23:45:07.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:07 smithi178 ceph-mon[47547]: osdmap e57: 9 total, 9 up, 9 in 2024-02-21T23:45:07.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:07 smithi090 ceph-mon[43385]: purged_snaps scrub starts 2024-02-21T23:45:07.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:07 smithi090 ceph-mon[43385]: purged_snaps scrub ok 2024-02-21T23:45:07.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:07 smithi090 ceph-mon[43385]: from='client.14433 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:45:07.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:07 smithi090 ceph-mon[43385]: osd.8 [v2:172.21.15.178:6816/490229963,v1:172.21.15.178:6817/490229963] boot 2024-02-21T23:45:07.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:07 smithi090 ceph-mon[43385]: osdmap e56: 9 total, 9 up, 9 in 2024-02-21T23:45:07.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:07 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-21T23:45:07.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:07 smithi090 ceph-mon[43385]: osdmap e57: 9 total, 9 up, 9 in 2024-02-21T23:45:08.583 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:08 smithi176 ceph-mon[47172]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2024-02-21T23:45:08.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:08 smithi176 ceph-mon[47172]: osdmap e58: 9 total, 9 up, 9 in 2024-02-21T23:45:08.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:08 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/3497166592' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c2cecd4-55de-4fc4-a6f4-ca97c8460444"}]: dispatch 2024-02-21T23:45:08.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:08 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/3497166592' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0c2cecd4-55de-4fc4-a6f4-ca97c8460444"}]': finished 2024-02-21T23:45:08.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:08 smithi176 ceph-mon[47172]: osdmap e59: 10 total, 9 up, 10 in 2024-02-21T23:45:08.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:08 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-21T23:45:08.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:08 smithi178 ceph-mon[47547]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2024-02-21T23:45:08.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:08 smithi178 ceph-mon[47547]: osdmap e58: 9 total, 9 up, 9 in 2024-02-21T23:45:08.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:08 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/3497166592' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c2cecd4-55de-4fc4-a6f4-ca97c8460444"}]: dispatch 2024-02-21T23:45:08.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:08 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/3497166592' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0c2cecd4-55de-4fc4-a6f4-ca97c8460444"}]': finished 2024-02-21T23:45:08.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:08 smithi178 ceph-mon[47547]: osdmap e59: 10 total, 9 up, 10 in 2024-02-21T23:45:08.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:08 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-21T23:45:08.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:08 smithi090 ceph-mon[43385]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2024-02-21T23:45:08.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:08 smithi090 ceph-mon[43385]: osdmap e58: 9 total, 9 up, 9 in 2024-02-21T23:45:08.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:08 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/3497166592' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c2cecd4-55de-4fc4-a6f4-ca97c8460444"}]: dispatch 2024-02-21T23:45:08.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:08 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/3497166592' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0c2cecd4-55de-4fc4-a6f4-ca97c8460444"}]': finished 2024-02-21T23:45:08.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:08 smithi090 ceph-mon[43385]: osdmap e59: 10 total, 9 up, 10 in 2024-02-21T23:45:08.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:08 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-21T23:45:09.583 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:09 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/3576689944' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-21T23:45:09.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:09 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/3576689944' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-21T23:45:09.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:09 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/3576689944' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-21T23:45:10.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:10 smithi176 ceph-mon[47172]: osdmap e60: 10 total, 9 up, 10 in 2024-02-21T23:45:10.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:10 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-21T23:45:10.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:10 smithi176 ceph-mon[47172]: pgmap v158: 1 pgs: 1 active+recovery_wait+undersized+degraded+remapped; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 2/6 objects degraded (33.333%) 2024-02-21T23:45:10.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:10 smithi178 ceph-mon[47547]: osdmap e60: 10 total, 9 up, 10 in 2024-02-21T23:45:10.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:10 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-21T23:45:10.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:10 smithi178 ceph-mon[47547]: pgmap v158: 1 pgs: 1 active+recovery_wait+undersized+degraded+remapped; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 2/6 objects degraded (33.333%) 2024-02-21T23:45:10.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:10 smithi090 ceph-mon[43385]: osdmap e60: 10 total, 9 up, 10 in 2024-02-21T23:45:10.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:10 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-21T23:45:10.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:10 smithi090 ceph-mon[43385]: pgmap v158: 1 pgs: 1 active+recovery_wait+undersized+degraded+remapped; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 2/6 objects degraded (33.333%) 2024-02-21T23:45:11.583 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:11 smithi176 ceph-mon[47172]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-02-21T23:45:11.583 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:11 smithi176 ceph-mon[47172]: osdmap e61: 10 total, 9 up, 10 in 2024-02-21T23:45:11.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:11 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-21T23:45:11.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:11 smithi178 ceph-mon[47547]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-02-21T23:45:11.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:11 smithi178 ceph-mon[47547]: osdmap e61: 10 total, 9 up, 10 in 2024-02-21T23:45:11.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:11 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-21T23:45:11.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:11 smithi090 ceph-mon[43385]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-02-21T23:45:11.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:11 smithi090 ceph-mon[43385]: osdmap e61: 10 total, 9 up, 10 in 2024-02-21T23:45:11.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:11 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-21T23:45:12.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:12 smithi090 ceph-mon[43385]: pgmap v160: 1 pgs: 1 active+recovery_wait+undersized+degraded+remapped; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 2/6 objects degraded (33.333%) 2024-02-21T23:45:12.583 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:12 smithi176 ceph-mon[47172]: pgmap v160: 1 pgs: 1 active+recovery_wait+undersized+degraded+remapped; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 2/6 objects degraded (33.333%) 2024-02-21T23:45:12.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:12 smithi178 ceph-mon[47547]: pgmap v160: 1 pgs: 1 active+recovery_wait+undersized+degraded+remapped; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 2/6 objects degraded (33.333%) 2024-02-21T23:45:14.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:14 smithi090 ceph-mon[43385]: pgmap v161: 1 pgs: 1 active+recovery_wait+undersized+degraded+remapped; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 2/6 objects degraded (33.333%) 2024-02-21T23:45:14.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:14 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-02-21T23:45:14.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:14 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:14.833 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:14 smithi176 ceph-mon[47172]: pgmap v161: 1 pgs: 1 active+recovery_wait+undersized+degraded+remapped; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 2/6 objects degraded (33.333%) 2024-02-21T23:45:14.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:14 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-02-21T23:45:14.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:14 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:14.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:14 smithi178 ceph-mon[47547]: pgmap v161: 1 pgs: 1 active+recovery_wait+undersized+degraded+remapped; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 2/6 objects degraded (33.333%) 2024-02-21T23:45:14.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:14 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-02-21T23:45:14.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:14 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:15.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:15 smithi090 ceph-mon[43385]: Deploying daemon osd.9 on smithi090 2024-02-21T23:45:15.833 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:15 smithi176 ceph-mon[47172]: Deploying daemon osd.9 on smithi090 2024-02-21T23:45:15.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:15 smithi178 ceph-mon[47547]: Deploying daemon osd.9 on smithi090 2024-02-21T23:45:16.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:16 smithi090 ceph-mon[43385]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-02-21T23:45:16.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:16 smithi090 ceph-mon[43385]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-02-21T23:45:16.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:16 smithi090 ceph-mon[43385]: Cluster is now healthy 2024-02-21T23:45:16.833 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:16 smithi176 ceph-mon[47172]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-02-21T23:45:16.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:16 smithi176 ceph-mon[47172]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-02-21T23:45:16.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:16 smithi176 ceph-mon[47172]: Cluster is now healthy 2024-02-21T23:45:16.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:16 smithi178 ceph-mon[47547]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-02-21T23:45:16.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:16 smithi178 ceph-mon[47547]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-02-21T23:45:16.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:16 smithi178 ceph-mon[47547]: Cluster is now healthy 2024-02-21T23:45:17.953 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:17 smithi090 ceph-mon[43385]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-02-21T23:45:18.083 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:17 smithi176 ceph-mon[47172]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-02-21T23:45:18.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:17 smithi178 ceph-mon[47547]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-02-21T23:45:19.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:19 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:19.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:19 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:19.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:19 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:45:19.333 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:19 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:19.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:19 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:19.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:19 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:45:19.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:19 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:19.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:19 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:19.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:19 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:45:20.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:20 smithi090 ceph-mon[43385]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-21T23:45:20.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:20 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:20.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:20 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:20.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:20 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:20.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:20 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:45:20.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:20 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:20.333 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:20 smithi176 ceph-mon[47172]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-21T23:45:20.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:20 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:20.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:20 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:20.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:20 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:20.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:20 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:45:20.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:20 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:20.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:20 smithi178 ceph-mon[47547]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-21T23:45:20.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:20 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:20.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:20 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:20.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:20 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:20.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:20 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:45:20.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:20 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:21.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:21.054+0000 7fcc7882a700 1 -- 172.21.15.90:0/2561045598 <== mgr.14150 v2:172.21.15.90:6800/3697394685 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fcc7419f550 con 0x7fcc5c0a1120 2024-02-21T23:45:21.039 INFO:teuthology.orchestra.run.smithi090.stdout:Created osd(s) 9 on host 'smithi090' 2024-02-21T23:45:21.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:21.056+0000 7fcc7982c700 1 -- 172.21.15.90:0/2561045598 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fcc5c0a1120 msgr2=0x7fcc5c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:45:21.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:21.056+0000 7fcc7982c700 1 --2- 172.21.15.90:0/2561045598 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fcc5c0a1120 0x7fcc5c0a35e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fcc68009990 tx=0x7fcc68005c00 comp rx=0 tx=0).stop 2024-02-21T23:45:21.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:21.056+0000 7fcc7982c700 1 -- 172.21.15.90:0/2561045598 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc74100e90 msgr2=0x7fcc74197320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:45:21.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:21.056+0000 7fcc7982c700 1 --2- 172.21.15.90:0/2561045598 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc74100e90 0x7fcc74197320 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fcc6400b530 tx=0x7fcc6400b840 comp rx=0 tx=0).stop 2024-02-21T23:45:21.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:21.056+0000 7fcc7982c700 1 -- 172.21.15.90:0/2561045598 shutdown_connections 2024-02-21T23:45:21.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:21.056+0000 7fcc7982c700 1 --2- 172.21.15.90:0/2561045598 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fcc5c0a1120 0x7fcc5c0a35e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:21.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:21.056+0000 7fcc7982c700 1 --2- 172.21.15.90:0/2561045598 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcc74101830 0x7fcc7419e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:21.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:21.056+0000 7fcc7982c700 1 --2- 172.21.15.90:0/2561045598 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc740ffc90 0x7fcc74196de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:21.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:21.056+0000 7fcc7982c700 1 --2- 172.21.15.90:0/2561045598 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc74100e90 0x7fcc74197320 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:21.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:21.056+0000 7fcc7982c700 1 -- 172.21.15.90:0/2561045598 >> 172.21.15.90:0/2561045598 conn(0x7fcc740fb230 msgr2=0x7fcc74068790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:45:21.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:21.056+0000 7fcc7982c700 1 -- 172.21.15.90:0/2561045598 shutdown_connections 2024-02-21T23:45:21.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:21.056+0000 7fcc7982c700 1 -- 172.21.15.90:0/2561045598 wait complete. 2024-02-21T23:45:21.537 DEBUG:teuthology.orchestra.run.smithi090:osd.9> sudo journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@osd.9.service 2024-02-21T23:45:21.539 INFO:tasks.cephadm:Deploying osd.10 on smithi176 with /dev/vg_nvme/lv_1... 2024-02-21T23:45:21.540 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-21T23:45:21.571 INFO:journalctl@ceph.osd.9.smithi090.stdout:-- Logs begin at Wed 2024-02-21 23:24:49 UTC. -- 2024-02-21T23:45:21.784 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.b/config 2024-02-21T23:45:22.147 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:22 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:22.147 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:22 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:22.147 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:22 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:45:22.148 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:22 smithi176 ceph-mon[47172]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-21T23:45:22.151 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:45:22 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:45:22.062+0000 7f35f67cd640 -1 osd.9 0 log_to_monitors true 2024-02-21T23:45:22.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:22 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:22.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:22 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:22.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:22 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:45:22.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:22 smithi090 ceph-mon[43385]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-21T23:45:22.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:22 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:22.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:22 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:22.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:22 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:45:22.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:22 smithi178 ceph-mon[47547]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-21T23:45:23.052 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-21T23:45:23.052 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-21T23:45:23.052 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10+0 records in 2024-02-21T23:45:23.053 INFO:teuthology.orchestra.run.smithi176.stderr:10+0 records out 2024-02-21T23:45:23.053 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129578 s, 809 MB/s 2024-02-21T23:45:23.053 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping successful for: 2024-02-21T23:45:23.331 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:23 smithi176 ceph-mon[47172]: from='osd.9 [v2:172.21.15.90:6826/2061394761,v1:172.21.15.90:6827/2061394761]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-02-21T23:45:23.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:23 smithi178 ceph-mon[47547]: from='osd.9 [v2:172.21.15.90:6826/2061394761,v1:172.21.15.90:6827/2061394761]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-02-21T23:45:23.425 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph orch daemon add osd smithi176:vg_nvme/lv_1 2024-02-21T23:45:23.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:23 smithi090 ceph-mon[43385]: from='osd.9 [v2:172.21.15.90:6826/2061394761,v1:172.21.15.90:6827/2061394761]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-02-21T23:45:23.643 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.b/config 2024-02-21T23:45:24.227 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:45:24 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:45:24.038+0000 7f35ebe31700 -1 osd.9 0 waiting for initial osdmap 2024-02-21T23:45:24.227 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:45:24 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:45:24.050+0000 7f35e5c22700 -1 osd.9 63 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-21T23:45:24.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:24 smithi090 ceph-mon[43385]: from='osd.9 [v2:172.21.15.90:6826/2061394761,v1:172.21.15.90:6827/2061394761]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-02-21T23:45:24.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:24 smithi090 ceph-mon[43385]: osdmap e62: 10 total, 9 up, 10 in 2024-02-21T23:45:24.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:24 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-21T23:45:24.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:24 smithi090 ceph-mon[43385]: from='osd.9 [v2:172.21.15.90:6826/2061394761,v1:172.21.15.90:6827/2061394761]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-21T23:45:24.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:24 smithi090 ceph-mon[43385]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-21T23:45:24.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:24 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:24.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:24 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:24.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:24 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:45:24.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:24 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:45:24.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:24 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:45:24.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:24 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:45:24.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:24 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:24.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:24 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:24.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:24 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:45:24.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:24 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:24.333 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:24 smithi176 ceph-mon[47172]: from='osd.9 [v2:172.21.15.90:6826/2061394761,v1:172.21.15.90:6827/2061394761]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-02-21T23:45:24.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:24 smithi176 ceph-mon[47172]: osdmap e62: 10 total, 9 up, 10 in 2024-02-21T23:45:24.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:24 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-21T23:45:24.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:24 smithi176 ceph-mon[47172]: from='osd.9 [v2:172.21.15.90:6826/2061394761,v1:172.21.15.90:6827/2061394761]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-21T23:45:24.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:24 smithi176 ceph-mon[47172]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-21T23:45:24.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:24 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:24.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:24 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:24.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:24 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:45:24.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:24 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:45:24.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:24 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:45:24.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:24 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:45:24.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:24 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:24.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:24 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:24.335 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:24 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:45:24.335 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:24 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:24.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:24 smithi178 ceph-mon[47547]: from='osd.9 [v2:172.21.15.90:6826/2061394761,v1:172.21.15.90:6827/2061394761]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-02-21T23:45:24.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:24 smithi178 ceph-mon[47547]: osdmap e62: 10 total, 9 up, 10 in 2024-02-21T23:45:24.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:24 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-21T23:45:24.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:24 smithi178 ceph-mon[47547]: from='osd.9 [v2:172.21.15.90:6826/2061394761,v1:172.21.15.90:6827/2061394761]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-21T23:45:24.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:24 smithi178 ceph-mon[47547]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-21T23:45:24.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:24 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:24.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:24 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:24.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:24 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:45:24.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:24 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:45:24.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:24 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:45:24.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:24 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:45:24.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:24 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:24.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:24 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:24.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:24 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:45:24.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:24 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:24.618 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.631+0000 7f1e9392f700 1 -- 172.21.15.176:0/1475591274 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e8c101830 msgr2=0x7f1e8c1080e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:45:24.618 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.631+0000 7f1e9392f700 1 --2- 172.21.15.176:0/1475591274 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e8c101830 0x7f1e8c1080e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1e8400b0d0 tx=0x7f1e8400b3e0 comp rx=0 tx=0).stop 2024-02-21T23:45:24.618 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.631+0000 7f1e9392f700 1 -- 172.21.15.176:0/1475591274 shutdown_connections 2024-02-21T23:45:24.618 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.631+0000 7f1e9392f700 1 --2- 172.21.15.176:0/1475591274 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e8c101830 0x7f1e8c1080e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:24.618 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.631+0000 7f1e9392f700 1 --2- 172.21.15.176:0/1475591274 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1e8c0ffc90 0x7f1e8c1000b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:24.618 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.631+0000 7f1e9392f700 1 --2- 172.21.15.176:0/1475591274 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e8c100e90 0x7f1e8c1012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:24.618 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.631+0000 7f1e9392f700 1 -- 172.21.15.176:0/1475591274 >> 172.21.15.176:0/1475591274 conn(0x7f1e8c0fb230 msgr2=0x7f1e8c0fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:45:24.618 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.631+0000 7f1e9392f700 1 -- 172.21.15.176:0/1475591274 shutdown_connections 2024-02-21T23:45:24.618 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.631+0000 7f1e9392f700 1 -- 172.21.15.176:0/1475591274 wait complete. 2024-02-21T23:45:24.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.632+0000 7f1e9392f700 1 Processor -- start 2024-02-21T23:45:24.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.632+0000 7f1e9392f700 1 -- start start 2024-02-21T23:45:24.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.632+0000 7f1e9392f700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1e8c0ffc90 0x7f1e8c196df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:45:24.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.632+0000 7f1e9392f700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e8c100e90 0x7f1e8c197330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:45:24.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.632+0000 7f1e9392f700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e8c101830 0x7f1e8c19e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:45:24.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.632+0000 7f1e9392f700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c19e8e0 con 0x7f1e8c101830 2024-02-21T23:45:24.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.632+0000 7f1e9392f700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c19ea20 con 0x7f1e8c100e90 2024-02-21T23:45:24.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.632+0000 7f1e9392f700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c19ebb0 con 0x7f1e8c0ffc90 2024-02-21T23:45:24.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.632+0000 7f1e90eca700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e8c100e90 0x7f1e8c197330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:45:24.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.632+0000 7f1e90eca700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e8c100e90 0x7f1e8c197330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:36060/0 (socket says 172.21.15.176:36060) 2024-02-21T23:45:24.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.632+0000 7f1e90eca700 1 -- 172.21.15.176:0/565527443 learned_addr learned my addr 172.21.15.176:0/565527443 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-21T23:45:24.620 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.633+0000 7f1e916cb700 1 --2- 172.21.15.176:0/565527443 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1e8c0ffc90 0x7f1e8c196df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:45:24.620 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.633+0000 7f1e90eca700 1 -- 172.21.15.176:0/565527443 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1e8c0ffc90 msgr2=0x7f1e8c196df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:45:24.620 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.633+0000 7f1e91ecc700 1 --2- 172.21.15.176:0/565527443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e8c101830 0x7f1e8c19e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:45:24.620 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.633+0000 7f1e90eca700 1 --2- 172.21.15.176:0/565527443 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1e8c0ffc90 0x7f1e8c196df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:24.620 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.633+0000 7f1e90eca700 1 -- 172.21.15.176:0/565527443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e8c101830 msgr2=0x7f1e8c19e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:45:24.620 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.633+0000 7f1e90eca700 1 --2- 172.21.15.176:0/565527443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e8c101830 0x7f1e8c19e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:24.620 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.633+0000 7f1e90eca700 1 -- 172.21.15.176:0/565527443 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e84009d00 con 0x7f1e8c100e90 2024-02-21T23:45:24.620 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.633+0000 7f1e916cb700 1 --2- 172.21.15.176:0/565527443 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1e8c0ffc90 0x7f1e8c196df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:45:24.620 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.633+0000 7f1e90eca700 1 --2- 172.21.15.176:0/565527443 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e8c100e90 0x7f1e8c197330 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1e8800e9a0 tx=0x7f1e8800ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:45:24.620 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.633+0000 7f1e827fc700 1 -- 172.21.15.176:0/565527443 <== mon.2 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e88009790 con 0x7f1e8c100e90 2024-02-21T23:45:24.620 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.633+0000 7f1e9392f700 1 -- 172.21.15.176:0/565527443 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e8c19ee90 con 0x7f1e8c100e90 2024-02-21T23:45:24.620 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.633+0000 7f1e827fc700 1 -- 172.21.15.176:0/565527443 <== mon.2 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1e8800cbd0 con 0x7f1e8c100e90 2024-02-21T23:45:24.620 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.633+0000 7f1e827fc700 1 -- 172.21.15.176:0/565527443 <== mon.2 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e88018680 con 0x7f1e8c100e90 2024-02-21T23:45:24.621 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.633+0000 7f1e9392f700 1 -- 172.21.15.176:0/565527443 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e8c19f300 con 0x7f1e8c100e90 2024-02-21T23:45:24.621 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.634+0000 7f1e9392f700 1 -- 172.21.15.176:0/565527443 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e8c04ea40 con 0x7f1e8c100e90 2024-02-21T23:45:24.621 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.635+0000 7f1e827fc700 1 -- 172.21.15.176:0/565527443 <== mon.2 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7f1e8800cd40 con 0x7f1e8c100e90 2024-02-21T23:45:24.622 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.635+0000 7f1e827fc700 1 --2- 172.21.15.176:0/565527443 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f1e780a1120 0x7f1e780a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:45:24.622 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.636+0000 7f1e827fc700 1 -- 172.21.15.176:0/565527443 <== mon.2 v2:172.21.15.176:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5981+0+0 (secure 0 0 0) 0x7f1e88014070 con 0x7f1e8c100e90 2024-02-21T23:45:24.622 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.636+0000 7f1e916cb700 1 --2- 172.21.15.176:0/565527443 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f1e780a1120 0x7f1e780a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:45:24.622 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.636+0000 7f1e916cb700 1 --2- 172.21.15.176:0/565527443 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f1e780a1120 0x7f1e780a35e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c005fd0 tx=0x7f1e7c005f00 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:45:24.623 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.640+0000 7f1e827fc700 1 -- 172.21.15.176:0/565527443 <== mon.2 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f1e880963a0 con 0x7f1e8c100e90 2024-02-21T23:45:24.822 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:24.838+0000 7f1e9392f700 1 -- 172.21.15.176:0/565527443 --> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f1e8c0fd3f0 con 0x7f1e780a1120 2024-02-21T23:45:25.327 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:25 smithi176 ceph-mon[47172]: Detected new or changed devices on smithi090 2024-02-21T23:45:25.327 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:25 smithi176 ceph-mon[47172]: Adjusting osd_memory_target on smithi090 to 4278M 2024-02-21T23:45:25.328 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:25 smithi176 ceph-mon[47172]: from='osd.9 [v2:172.21.15.90:6826/2061394761,v1:172.21.15.90:6827/2061394761]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-02-21T23:45:25.328 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:25 smithi176 ceph-mon[47172]: osdmap e63: 10 total, 9 up, 10 in 2024-02-21T23:45:25.328 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:25 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-21T23:45:25.328 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:25 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-21T23:45:25.328 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:25 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-21T23:45:25.328 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:25 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-21T23:45:25.328 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:25 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:25.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:25 smithi178 ceph-mon[47547]: Detected new or changed devices on smithi090 2024-02-21T23:45:25.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:25 smithi178 ceph-mon[47547]: Adjusting osd_memory_target on smithi090 to 4278M 2024-02-21T23:45:25.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:25 smithi178 ceph-mon[47547]: from='osd.9 [v2:172.21.15.90:6826/2061394761,v1:172.21.15.90:6827/2061394761]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-02-21T23:45:25.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:25 smithi178 ceph-mon[47547]: osdmap e63: 10 total, 9 up, 10 in 2024-02-21T23:45:25.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:25 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-21T23:45:25.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:25 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-21T23:45:25.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:25 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-21T23:45:25.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:25 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-21T23:45:25.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:25 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:25.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:25 smithi090 ceph-mon[43385]: Detected new or changed devices on smithi090 2024-02-21T23:45:25.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:25 smithi090 ceph-mon[43385]: Adjusting osd_memory_target on smithi090 to 4278M 2024-02-21T23:45:25.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:25 smithi090 ceph-mon[43385]: from='osd.9 [v2:172.21.15.90:6826/2061394761,v1:172.21.15.90:6827/2061394761]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-02-21T23:45:25.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:25 smithi090 ceph-mon[43385]: osdmap e63: 10 total, 9 up, 10 in 2024-02-21T23:45:25.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:25 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-21T23:45:25.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:25 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-21T23:45:25.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:25 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-21T23:45:25.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:25 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-21T23:45:25.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:25 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:26.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:26 smithi176 ceph-mon[47172]: purged_snaps scrub starts 2024-02-21T23:45:26.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:26 smithi176 ceph-mon[47172]: purged_snaps scrub ok 2024-02-21T23:45:26.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:26 smithi176 ceph-mon[47172]: from='client.24332 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:45:26.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:26 smithi176 ceph-mon[47172]: osd.9 [v2:172.21.15.90:6826/2061394761,v1:172.21.15.90:6827/2061394761] boot 2024-02-21T23:45:26.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:26 smithi176 ceph-mon[47172]: osdmap e64: 10 total, 10 up, 10 in 2024-02-21T23:45:26.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:26 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-21T23:45:26.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:26 smithi176 ceph-mon[47172]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-02-21T23:45:26.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:26 smithi178 ceph-mon[47547]: purged_snaps scrub starts 2024-02-21T23:45:26.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:26 smithi178 ceph-mon[47547]: purged_snaps scrub ok 2024-02-21T23:45:26.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:26 smithi178 ceph-mon[47547]: from='client.24332 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:45:26.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:26 smithi178 ceph-mon[47547]: osd.9 [v2:172.21.15.90:6826/2061394761,v1:172.21.15.90:6827/2061394761] boot 2024-02-21T23:45:26.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:26 smithi178 ceph-mon[47547]: osdmap e64: 10 total, 10 up, 10 in 2024-02-21T23:45:26.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:26 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-21T23:45:26.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:26 smithi178 ceph-mon[47547]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-02-21T23:45:26.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:26 smithi090 ceph-mon[43385]: purged_snaps scrub starts 2024-02-21T23:45:26.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:26 smithi090 ceph-mon[43385]: purged_snaps scrub ok 2024-02-21T23:45:26.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:26 smithi090 ceph-mon[43385]: from='client.24332 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:45:26.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:26 smithi090 ceph-mon[43385]: osd.9 [v2:172.21.15.90:6826/2061394761,v1:172.21.15.90:6827/2061394761] boot 2024-02-21T23:45:26.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:26 smithi090 ceph-mon[43385]: osdmap e64: 10 total, 10 up, 10 in 2024-02-21T23:45:26.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:26 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-21T23:45:26.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:26 smithi090 ceph-mon[43385]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-02-21T23:45:27.333 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:27 smithi176 ceph-mon[47172]: osdmap e65: 10 total, 10 up, 10 in 2024-02-21T23:45:27.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:27 smithi176 ceph-mon[47172]: from='client.? 172.21.15.176:0/2075140855' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa04232b-b1ec-447a-8e94-7c9bf26d0291"}]: dispatch 2024-02-21T23:45:27.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:27 smithi176 ceph-mon[47172]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa04232b-b1ec-447a-8e94-7c9bf26d0291"}]: dispatch 2024-02-21T23:45:27.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:27 smithi176 ceph-mon[47172]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fa04232b-b1ec-447a-8e94-7c9bf26d0291"}]': finished 2024-02-21T23:45:27.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:27 smithi176 ceph-mon[47172]: osdmap e66: 11 total, 10 up, 11 in 2024-02-21T23:45:27.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:27 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-21T23:45:27.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:27 smithi178 ceph-mon[47547]: osdmap e65: 10 total, 10 up, 10 in 2024-02-21T23:45:27.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:27 smithi178 ceph-mon[47547]: from='client.? 172.21.15.176:0/2075140855' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa04232b-b1ec-447a-8e94-7c9bf26d0291"}]: dispatch 2024-02-21T23:45:27.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:27 smithi178 ceph-mon[47547]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa04232b-b1ec-447a-8e94-7c9bf26d0291"}]: dispatch 2024-02-21T23:45:27.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:27 smithi178 ceph-mon[47547]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fa04232b-b1ec-447a-8e94-7c9bf26d0291"}]': finished 2024-02-21T23:45:27.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:27 smithi178 ceph-mon[47547]: osdmap e66: 11 total, 10 up, 11 in 2024-02-21T23:45:27.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:27 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-21T23:45:27.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:27 smithi090 ceph-mon[43385]: osdmap e65: 10 total, 10 up, 10 in 2024-02-21T23:45:27.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:27 smithi090 ceph-mon[43385]: from='client.? 172.21.15.176:0/2075140855' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa04232b-b1ec-447a-8e94-7c9bf26d0291"}]: dispatch 2024-02-21T23:45:27.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:27 smithi090 ceph-mon[43385]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa04232b-b1ec-447a-8e94-7c9bf26d0291"}]: dispatch 2024-02-21T23:45:27.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:27 smithi090 ceph-mon[43385]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fa04232b-b1ec-447a-8e94-7c9bf26d0291"}]': finished 2024-02-21T23:45:27.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:27 smithi090 ceph-mon[43385]: osdmap e66: 11 total, 10 up, 11 in 2024-02-21T23:45:27.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:27 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-21T23:45:28.333 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:28 smithi176 ceph-mon[47172]: from='client.? 172.21.15.176:0/3489904079' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-21T23:45:28.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:28 smithi176 ceph-mon[47172]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-02-21T23:45:28.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:28 smithi178 ceph-mon[47547]: from='client.? 172.21.15.176:0/3489904079' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-21T23:45:28.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:28 smithi178 ceph-mon[47547]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-02-21T23:45:28.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:28 smithi090 ceph-mon[43385]: from='client.? 172.21.15.176:0/3489904079' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-21T23:45:28.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:28 smithi090 ceph-mon[43385]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-02-21T23:45:30.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:30 smithi090 ceph-mon[43385]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-02-21T23:45:30.833 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:30 smithi176 ceph-mon[47172]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-02-21T23:45:30.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:30 smithi178 ceph-mon[47547]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-02-21T23:45:32.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:32 smithi090 ceph-mon[43385]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-02-21T23:45:32.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:32 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-02-21T23:45:32.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:32 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:32.800 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:32 smithi176 ceph-mon[47172]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-02-21T23:45:32.800 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:32 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-02-21T23:45:32.800 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:32 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:32.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:32 smithi178 ceph-mon[47547]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-02-21T23:45:32.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:32 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-02-21T23:45:32.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:32 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:33.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:33 smithi090 ceph-mon[43385]: Deploying daemon osd.10 on smithi176 2024-02-21T23:45:33.833 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:33 smithi176 ceph-mon[47172]: Deploying daemon osd.10 on smithi176 2024-02-21T23:45:33.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:33 smithi178 ceph-mon[47547]: Deploying daemon osd.10 on smithi176 2024-02-21T23:45:34.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:34 smithi090 ceph-mon[43385]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-02-21T23:45:34.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:34 smithi176 ceph-mon[47172]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-02-21T23:45:34.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:34 smithi178 ceph-mon[47547]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-02-21T23:45:36.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:36 smithi090 ceph-mon[43385]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-02-21T23:45:36.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:36 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:36.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:36 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:36.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:36 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:45:36.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:36 smithi176 ceph-mon[47172]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-02-21T23:45:36.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:36 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:36.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:36 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:36.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:36 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:45:36.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:36 smithi178 ceph-mon[47547]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-02-21T23:45:36.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:36 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:36.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:36 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:36.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:36 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:45:38.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:37 smithi176 ceph-mon[47172]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-02-21T23:45:38.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:37 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:38.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:37 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:38.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:37 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:38.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:37 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:45:38.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:37 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:38.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:37 smithi178 ceph-mon[47547]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-02-21T23:45:38.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:37 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:38.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:37 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:38.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:37 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:38.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:37 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:45:38.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:37 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:38.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:37 smithi090 ceph-mon[43385]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-02-21T23:45:38.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:37 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:38.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:37 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:38.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:37 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:38.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:37 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:45:38.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:37 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:38.273 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:38.289+0000 7f1e827fc700 1 -- 172.21.15.176:0/565527443 <== mgr.14150 v2:172.21.15.90:6800/3697394685 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f1e8c0fd3f0 con 0x7f1e780a1120 2024-02-21T23:45:38.273 INFO:teuthology.orchestra.run.smithi176.stdout:Created osd(s) 10 on host 'smithi176' 2024-02-21T23:45:38.275 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:38.291+0000 7f1e9392f700 1 -- 172.21.15.176:0/565527443 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f1e780a1120 msgr2=0x7f1e780a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:45:38.275 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:38.291+0000 7f1e9392f700 1 --2- 172.21.15.176:0/565527443 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f1e780a1120 0x7f1e780a35e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c005fd0 tx=0x7f1e7c005f00 comp rx=0 tx=0).stop 2024-02-21T23:45:38.276 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:38.291+0000 7f1e9392f700 1 -- 172.21.15.176:0/565527443 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e8c100e90 msgr2=0x7f1e8c197330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:45:38.276 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:38.291+0000 7f1e9392f700 1 --2- 172.21.15.176:0/565527443 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e8c100e90 0x7f1e8c197330 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1e8800e9a0 tx=0x7f1e8800ed60 comp rx=0 tx=0).stop 2024-02-21T23:45:38.276 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:38.291+0000 7f1e9392f700 1 -- 172.21.15.176:0/565527443 shutdown_connections 2024-02-21T23:45:38.276 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:38.291+0000 7f1e9392f700 1 --2- 172.21.15.176:0/565527443 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f1e780a1120 0x7f1e780a35e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:38.276 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:38.291+0000 7f1e9392f700 1 --2- 172.21.15.176:0/565527443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e8c101830 0x7f1e8c19e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:38.276 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:38.291+0000 7f1e9392f700 1 --2- 172.21.15.176:0/565527443 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1e8c0ffc90 0x7f1e8c196df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:38.276 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:38.291+0000 7f1e9392f700 1 --2- 172.21.15.176:0/565527443 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e8c100e90 0x7f1e8c197330 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:38.276 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:38.291+0000 7f1e9392f700 1 -- 172.21.15.176:0/565527443 >> 172.21.15.176:0/565527443 conn(0x7f1e8c0fb230 msgr2=0x7f1e8c0fcce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:45:38.276 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:38.291+0000 7f1e9392f700 1 -- 172.21.15.176:0/565527443 shutdown_connections 2024-02-21T23:45:38.276 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-21T23:45:38.291+0000 7f1e9392f700 1 -- 172.21.15.176:0/565527443 wait complete. 2024-02-21T23:45:38.661 DEBUG:teuthology.orchestra.run.smithi176:osd.10> sudo journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@osd.10.service 2024-02-21T23:45:38.664 INFO:tasks.cephadm:Deploying osd.11 on smithi178 with /dev/vg_nvme/lv_1... 2024-02-21T23:45:38.664 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-21T23:45:38.694 INFO:journalctl@ceph.osd.10.smithi176.stdout:-- Logs begin at Wed 2024-02-21 23:23:58 UTC. -- 2024-02-21T23:45:38.895 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.c/config 2024-02-21T23:45:39.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:39 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:39.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:39 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:45:39.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:39 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:39.428 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:39 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:39.428 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:39 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:45:39.428 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:39 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:39.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:39 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:39.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:39 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:45:39.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:39 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:40.384 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:45:40 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:45:40.136+0000 7f9110ed9640 -1 osd.10 0 log_to_monitors true 2024-02-21T23:45:40.384 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:40 smithi176 ceph-mon[47172]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-02-21T23:45:40.384 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:40 smithi176 ceph-mon[47172]: from='osd.10 [v2:172.21.15.176:6824/639411757,v1:172.21.15.176:6825/639411757]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-02-21T23:45:40.384 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:40 smithi176 ceph-mon[47172]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-02-21T23:45:40.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:40 smithi090 ceph-mon[43385]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-02-21T23:45:40.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:40 smithi090 ceph-mon[43385]: from='osd.10 [v2:172.21.15.176:6824/639411757,v1:172.21.15.176:6825/639411757]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-02-21T23:45:40.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:40 smithi090 ceph-mon[43385]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-02-21T23:45:40.483 INFO:teuthology.orchestra.run.smithi178.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-21T23:45:40.483 INFO:teuthology.orchestra.run.smithi178.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-21T23:45:40.484 INFO:teuthology.orchestra.run.smithi178.stderr: stderr: 10+0 records in 2024-02-21T23:45:40.484 INFO:teuthology.orchestra.run.smithi178.stderr:10+0 records out 2024-02-21T23:45:40.484 INFO:teuthology.orchestra.run.smithi178.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00841681 s, 1.2 GB/s 2024-02-21T23:45:40.484 INFO:teuthology.orchestra.run.smithi178.stderr:--> Zapping successful for: 2024-02-21T23:45:40.518 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:40 smithi178 ceph-mon[47547]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-02-21T23:45:40.518 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:40 smithi178 ceph-mon[47547]: from='osd.10 [v2:172.21.15.176:6824/639411757,v1:172.21.15.176:6825/639411757]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-02-21T23:45:40.518 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:40 smithi178 ceph-mon[47547]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-02-21T23:45:40.900 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph orch daemon add osd smithi178:vg_nvme/lv_1 2024-02-21T23:45:41.117 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.c/config 2024-02-21T23:45:41.584 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:45:41 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:45:41.249+0000 7f910653d700 -1 osd.10 0 waiting for initial osdmap 2024-02-21T23:45:41.584 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:45:41 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:45:41.265+0000 7f9100b2f700 -1 osd.10 68 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-21T23:45:41.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:41 smithi176 ceph-mon[47172]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-02-21T23:45:41.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:41 smithi176 ceph-mon[47172]: osdmap e67: 11 total, 10 up, 11 in 2024-02-21T23:45:41.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:41 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-21T23:45:41.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:41 smithi176 ceph-mon[47172]: from='osd.10 [v2:172.21.15.176:6824/639411757,v1:172.21.15.176:6825/639411757]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-21T23:45:41.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:41 smithi176 ceph-mon[47172]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-21T23:45:41.585 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:41 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:41.585 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:41 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:41.585 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:41 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:45:41.585 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:41 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:45:41.585 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:41 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:45:41.585 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:41 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:45:41.585 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:41 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:41.585 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:41 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:41.585 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:41 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:45:41.585 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:41 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:41.586 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:41 smithi178 ceph-mon[47547]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-02-21T23:45:41.586 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:41 smithi178 ceph-mon[47547]: osdmap e67: 11 total, 10 up, 11 in 2024-02-21T23:45:41.586 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:41 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-21T23:45:41.586 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:41 smithi178 ceph-mon[47547]: from='osd.10 [v2:172.21.15.176:6824/639411757,v1:172.21.15.176:6825/639411757]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-21T23:45:41.586 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:41 smithi178 ceph-mon[47547]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-21T23:45:41.586 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:41 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:41.586 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:41 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:41.586 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:41 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:45:41.586 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:41 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:45:41.586 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:41 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:45:41.586 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:41 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:45:41.587 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:41 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:41.587 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:41 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:41.587 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:41 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:45:41.587 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:41 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:41.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:41 smithi090 ceph-mon[43385]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-02-21T23:45:41.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:41 smithi090 ceph-mon[43385]: osdmap e67: 11 total, 10 up, 11 in 2024-02-21T23:45:41.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:41 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-21T23:45:41.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:41 smithi090 ceph-mon[43385]: from='osd.10 [v2:172.21.15.176:6824/639411757,v1:172.21.15.176:6825/639411757]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-21T23:45:41.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:41 smithi090 ceph-mon[43385]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-21T23:45:41.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:41 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:41.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:41 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:41.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:41 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:45:41.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:41 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:45:41.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:41 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:45:41.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:41 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:45:41.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:41 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:41.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:41 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:41.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:41 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:45:41.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:41 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:42.208 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.222+0000 7f94fcb16700 1 -- 172.21.15.178:0/2034535005 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f94f80fff00 msgr2=0x7f94f8100380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:45:42.208 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.222+0000 7f94fcb16700 1 --2- 172.21.15.178:0/2034535005 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f94f80fff00 0x7f94f8100380 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f94e8009990 tx=0x7f94e8009ca0 comp rx=0 tx=0).stop 2024-02-21T23:45:42.208 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.222+0000 7f94fcb16700 1 -- 172.21.15.178:0/2034535005 shutdown_connections 2024-02-21T23:45:42.208 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.222+0000 7f94fcb16700 1 --2- 172.21.15.178:0/2034535005 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94f8100910 0x7f94f8108160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:42.208 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.222+0000 7f94fcb16700 1 --2- 172.21.15.178:0/2034535005 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f94f8101430 0x7f94f8101850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:42.209 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.222+0000 7f94fcb16700 1 --2- 172.21.15.178:0/2034535005 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f94f80fff00 0x7f94f8100380 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:42.209 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.222+0000 7f94fcb16700 1 -- 172.21.15.178:0/2034535005 >> 172.21.15.178:0/2034535005 conn(0x7f94f80fa9d0 msgr2=0x7f94f80fce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:45:42.209 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.222+0000 7f94fcb16700 1 -- 172.21.15.178:0/2034535005 shutdown_connections 2024-02-21T23:45:42.209 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.222+0000 7f94fcb16700 1 -- 172.21.15.178:0/2034535005 wait complete. 2024-02-21T23:45:42.209 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.222+0000 7f94fcb16700 1 Processor -- start 2024-02-21T23:45:42.209 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.223+0000 7f94fcb16700 1 -- start start 2024-02-21T23:45:42.209 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.223+0000 7f94fcb16700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94f80fff00 0x7f94f8196e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:45:42.209 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.223+0000 7f94fcb16700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f94f8100910 0x7f94f8197340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:45:42.209 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.223+0000 7f94fcb16700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f94f8101430 0x7f94f819e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:45:42.209 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.223+0000 7f94fcb16700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f819e900 con 0x7f94f80fff00 2024-02-21T23:45:42.209 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.223+0000 7f94fcb16700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f819ea40 con 0x7f94f8100910 2024-02-21T23:45:42.209 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.223+0000 7f94fcb16700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f819ec00 con 0x7f94f8101430 2024-02-21T23:45:42.209 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.223+0000 7f94f6d9d700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f94f8101430 0x7f94f819e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:45:42.209 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.223+0000 7f94f6d9d700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f94f8101430 0x7f94f819e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:51220/0 (socket says 172.21.15.178:51220) 2024-02-21T23:45:42.210 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.223+0000 7f94f6d9d700 1 -- 172.21.15.178:0/3378985150 learned_addr learned my addr 172.21.15.178:0/3378985150 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-21T23:45:42.210 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.223+0000 7f94f5d9b700 1 --2- 172.21.15.178:0/3378985150 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f94f8100910 0x7f94f8197340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:45:42.210 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.223+0000 7f94f659c700 1 --2- 172.21.15.178:0/3378985150 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94f80fff00 0x7f94f8196e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:45:42.210 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.223+0000 7f94f6d9d700 1 -- 172.21.15.178:0/3378985150 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f94f8100910 msgr2=0x7f94f8197340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:45:42.210 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.223+0000 7f94f6d9d700 1 --2- 172.21.15.178:0/3378985150 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f94f8100910 0x7f94f8197340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:42.210 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.223+0000 7f94f6d9d700 1 -- 172.21.15.178:0/3378985150 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94f80fff00 msgr2=0x7f94f8196e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:45:42.210 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.223+0000 7f94f6d9d700 1 --2- 172.21.15.178:0/3378985150 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94f80fff00 0x7f94f8196e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:42.210 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.223+0000 7f94f6d9d700 1 -- 172.21.15.178:0/3378985150 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e80095f0 con 0x7f94f8101430 2024-02-21T23:45:42.210 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.223+0000 7f94f6d9d700 1 --2- 172.21.15.178:0/3378985150 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f94f8101430 0x7f94f819e330 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f94f000b970 tx=0x7f94f000bc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:45:42.210 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.223+0000 7f94e77fe700 1 -- 172.21.15.178:0/3378985150 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f000c930 con 0x7f94f8101430 2024-02-21T23:45:42.210 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.224+0000 7f94e77fe700 1 -- 172.21.15.178:0/3378985150 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f94f0014440 con 0x7f94f8101430 2024-02-21T23:45:42.210 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.224+0000 7f94fcb16700 1 -- 172.21.15.178:0/3378985150 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94f819eee0 con 0x7f94f8101430 2024-02-21T23:45:42.210 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.224+0000 7f94e77fe700 1 -- 172.21.15.178:0/3378985150 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f00075d0 con 0x7f94f8101430 2024-02-21T23:45:42.210 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.224+0000 7f94fcb16700 1 -- 172.21.15.178:0/3378985150 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94f819f2a0 con 0x7f94f8101430 2024-02-21T23:45:42.211 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.225+0000 7f94fcb16700 1 -- 172.21.15.178:0/3378985150 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94f804ea40 con 0x7f94f8101430 2024-02-21T23:45:42.211 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.226+0000 7f94e77fe700 1 -- 172.21.15.178:0/3378985150 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7f94f00145b0 con 0x7f94f8101430 2024-02-21T23:45:42.211 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.227+0000 7f94e77fe700 1 --2- 172.21.15.178:0/3378985150 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f94e00a1120 0x7f94e00a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:45:42.211 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.227+0000 7f94e77fe700 1 -- 172.21.15.178:0/3378985150 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6727+0+0 (secure 0 0 0) 0x7f94f0006e40 con 0x7f94f8101430 2024-02-21T23:45:42.212 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.227+0000 7f94f659c700 1 --2- 172.21.15.178:0/3378985150 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f94e00a1120 0x7f94e00a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:45:42.212 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.227+0000 7f94f659c700 1 --2- 172.21.15.178:0/3378985150 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f94e00a1120 0x7f94e00a35e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f94ec005950 tx=0x7f94ec005ed0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:45:42.214 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.230+0000 7f94e77fe700 1 -- 172.21.15.178:0/3378985150 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f94f0095b60 con 0x7f94f8101430 2024-02-21T23:45:42.419 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:42.435+0000 7f94fcb16700 1 -- 172.21.15.178:0/3378985150 --> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f94f819f4f0 con 0x7f94e00a1120 2024-02-21T23:45:42.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:42 smithi178 ceph-mon[47547]: Detected new or changed devices on smithi176 2024-02-21T23:45:42.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:42 smithi178 ceph-mon[47547]: Adjusting osd_memory_target on smithi176 to 4278M 2024-02-21T23:45:42.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:42 smithi178 ceph-mon[47547]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-02-21T23:45:42.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:42 smithi178 ceph-mon[47547]: osdmap e68: 11 total, 10 up, 11 in 2024-02-21T23:45:42.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:42 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-21T23:45:42.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:42 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-21T23:45:42.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:42 smithi178 ceph-mon[47547]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-02-21T23:45:42.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:42 smithi178 ceph-mon[47547]: osd.10 [v2:172.21.15.176:6824/639411757,v1:172.21.15.176:6825/639411757] boot 2024-02-21T23:45:42.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:42 smithi178 ceph-mon[47547]: osdmap e69: 11 total, 11 up, 11 in 2024-02-21T23:45:42.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:42 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-21T23:45:42.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:42 smithi176 ceph-mon[47172]: Detected new or changed devices on smithi176 2024-02-21T23:45:42.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:42 smithi176 ceph-mon[47172]: Adjusting osd_memory_target on smithi176 to 4278M 2024-02-21T23:45:42.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:42 smithi176 ceph-mon[47172]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-02-21T23:45:42.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:42 smithi176 ceph-mon[47172]: osdmap e68: 11 total, 10 up, 11 in 2024-02-21T23:45:42.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:42 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-21T23:45:42.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:42 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-21T23:45:42.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:42 smithi176 ceph-mon[47172]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-02-21T23:45:42.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:42 smithi176 ceph-mon[47172]: osd.10 [v2:172.21.15.176:6824/639411757,v1:172.21.15.176:6825/639411757] boot 2024-02-21T23:45:42.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:42 smithi176 ceph-mon[47172]: osdmap e69: 11 total, 11 up, 11 in 2024-02-21T23:45:42.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:42 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-21T23:45:42.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:42 smithi090 ceph-mon[43385]: Detected new or changed devices on smithi176 2024-02-21T23:45:42.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:42 smithi090 ceph-mon[43385]: Adjusting osd_memory_target on smithi176 to 4278M 2024-02-21T23:45:42.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:42 smithi090 ceph-mon[43385]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-02-21T23:45:42.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:42 smithi090 ceph-mon[43385]: osdmap e68: 11 total, 10 up, 11 in 2024-02-21T23:45:42.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:42 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-21T23:45:42.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:42 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-21T23:45:42.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:42 smithi090 ceph-mon[43385]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-02-21T23:45:42.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:42 smithi090 ceph-mon[43385]: osd.10 [v2:172.21.15.176:6824/639411757,v1:172.21.15.176:6825/639411757] boot 2024-02-21T23:45:42.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:42 smithi090 ceph-mon[43385]: osdmap e69: 11 total, 11 up, 11 in 2024-02-21T23:45:42.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:42 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-21T23:45:43.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:43 smithi176 ceph-mon[47172]: purged_snaps scrub starts 2024-02-21T23:45:43.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:43 smithi176 ceph-mon[47172]: purged_snaps scrub ok 2024-02-21T23:45:43.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:43 smithi176 ceph-mon[47172]: from='client.24337 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:45:43.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:43 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-21T23:45:43.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:43 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-21T23:45:43.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:43 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:43.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:43 smithi176 ceph-mon[47172]: osdmap e70: 11 total, 11 up, 11 in 2024-02-21T23:45:43.594 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:43 smithi178 ceph-mon[47547]: purged_snaps scrub starts 2024-02-21T23:45:43.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:43 smithi178 ceph-mon[47547]: purged_snaps scrub ok 2024-02-21T23:45:43.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:43 smithi178 ceph-mon[47547]: from='client.24337 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:45:43.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:43 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-21T23:45:43.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:43 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-21T23:45:43.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:43 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:43.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:43 smithi178 ceph-mon[47547]: osdmap e70: 11 total, 11 up, 11 in 2024-02-21T23:45:43.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:43 smithi090 ceph-mon[43385]: purged_snaps scrub starts 2024-02-21T23:45:43.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:43 smithi090 ceph-mon[43385]: purged_snaps scrub ok 2024-02-21T23:45:43.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:43 smithi090 ceph-mon[43385]: from='client.24337 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:45:43.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:43 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-21T23:45:43.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:43 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-21T23:45:43.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:43 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:43.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:43 smithi090 ceph-mon[43385]: osdmap e70: 11 total, 11 up, 11 in 2024-02-21T23:45:44.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:44 smithi176 ceph-mon[47172]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-02-21T23:45:44.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:44 smithi176 ceph-mon[47172]: osdmap e71: 11 total, 11 up, 11 in 2024-02-21T23:45:44.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:44 smithi178 ceph-mon[47547]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-02-21T23:45:44.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:44 smithi178 ceph-mon[47547]: osdmap e71: 11 total, 11 up, 11 in 2024-02-21T23:45:44.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:44 smithi090 ceph-mon[43385]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-02-21T23:45:44.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:44 smithi090 ceph-mon[43385]: osdmap e71: 11 total, 11 up, 11 in 2024-02-21T23:45:45.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:45 smithi176 ceph-mon[47172]: from='client.? 172.21.15.178:0/922320614' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7282e294-64fc-4856-9c74-86115f69408c"}]: dispatch 2024-02-21T23:45:45.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:45 smithi176 ceph-mon[47172]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7282e294-64fc-4856-9c74-86115f69408c"}]: dispatch 2024-02-21T23:45:45.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:45 smithi176 ceph-mon[47172]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7282e294-64fc-4856-9c74-86115f69408c"}]': finished 2024-02-21T23:45:45.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:45 smithi176 ceph-mon[47172]: osdmap e72: 12 total, 11 up, 12 in 2024-02-21T23:45:45.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:45 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-21T23:45:45.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:45 smithi176 ceph-mon[47172]: from='client.? 172.21.15.178:0/1151504811' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-21T23:45:45.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:45 smithi178 ceph-mon[47547]: from='client.? 172.21.15.178:0/922320614' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7282e294-64fc-4856-9c74-86115f69408c"}]: dispatch 2024-02-21T23:45:45.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:45 smithi178 ceph-mon[47547]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7282e294-64fc-4856-9c74-86115f69408c"}]: dispatch 2024-02-21T23:45:45.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:45 smithi178 ceph-mon[47547]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7282e294-64fc-4856-9c74-86115f69408c"}]': finished 2024-02-21T23:45:45.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:45 smithi178 ceph-mon[47547]: osdmap e72: 12 total, 11 up, 12 in 2024-02-21T23:45:45.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:45 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-21T23:45:45.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:45 smithi178 ceph-mon[47547]: from='client.? 172.21.15.178:0/1151504811' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-21T23:45:45.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:45 smithi090 ceph-mon[43385]: from='client.? 172.21.15.178:0/922320614' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7282e294-64fc-4856-9c74-86115f69408c"}]: dispatch 2024-02-21T23:45:45.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:45 smithi090 ceph-mon[43385]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7282e294-64fc-4856-9c74-86115f69408c"}]: dispatch 2024-02-21T23:45:45.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:45 smithi090 ceph-mon[43385]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7282e294-64fc-4856-9c74-86115f69408c"}]': finished 2024-02-21T23:45:45.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:45 smithi090 ceph-mon[43385]: osdmap e72: 12 total, 11 up, 12 in 2024-02-21T23:45:45.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:45 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-21T23:45:45.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:45 smithi090 ceph-mon[43385]: from='client.? 172.21.15.178:0/1151504811' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-21T23:45:46.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:46 smithi178 ceph-mon[47547]: pgmap v188: 1 pgs: 1 remapped+peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-02-21T23:45:46.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:46 smithi178 ceph-mon[47547]: osdmap e73: 12 total, 11 up, 12 in 2024-02-21T23:45:46.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:46 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-21T23:45:46.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:46 smithi090 ceph-mon[43385]: pgmap v188: 1 pgs: 1 remapped+peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-02-21T23:45:46.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:46 smithi090 ceph-mon[43385]: osdmap e73: 12 total, 11 up, 12 in 2024-02-21T23:45:46.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:46 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-21T23:45:46.833 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:46 smithi176 ceph-mon[47172]: pgmap v188: 1 pgs: 1 remapped+peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-02-21T23:45:46.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:46 smithi176 ceph-mon[47172]: osdmap e73: 12 total, 11 up, 12 in 2024-02-21T23:45:46.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:46 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-21T23:45:48.076 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:47 smithi178 ceph-mon[47547]: pgmap v190: 1 pgs: 1 remapped+peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-02-21T23:45:48.083 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:47 smithi176 ceph-mon[47172]: pgmap v190: 1 pgs: 1 remapped+peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-02-21T23:45:48.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:47 smithi090 ceph-mon[43385]: pgmap v190: 1 pgs: 1 remapped+peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-02-21T23:45:50.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:50 smithi178 ceph-mon[47547]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-21T23:45:50.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:50 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-02-21T23:45:50.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:50 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:50.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:50 smithi090 ceph-mon[43385]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-21T23:45:50.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:50 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-02-21T23:45:50.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:50 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:50.833 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:50 smithi176 ceph-mon[47172]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-21T23:45:50.833 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:50 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-02-21T23:45:50.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:50 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:51.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:51 smithi178 ceph-mon[47547]: Deploying daemon osd.11 on smithi178 2024-02-21T23:45:51.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:51 smithi090 ceph-mon[43385]: Deploying daemon osd.11 on smithi178 2024-02-21T23:45:51.833 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:51 smithi176 ceph-mon[47172]: Deploying daemon osd.11 on smithi178 2024-02-21T23:45:52.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:52 smithi178 ceph-mon[47547]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-02-21T23:45:52.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:52 smithi090 ceph-mon[43385]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-02-21T23:45:52.833 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:52 smithi176 ceph-mon[47172]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-02-21T23:45:54.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:54 smithi178 ceph-mon[47547]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-02-21T23:45:54.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:54 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:54.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:54 smithi090 ceph-mon[43385]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-02-21T23:45:54.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:54 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:54.833 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:54 smithi176 ceph-mon[47172]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-02-21T23:45:54.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:54 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:55.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:55 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:55.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:55 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:45:55.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:55 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:55.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:55 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:45:55.833 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:55 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:55.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:55 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:45:56.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:56 smithi090 ceph-mon[43385]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-21T23:45:56.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:56 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:56.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:56 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:56.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:56 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:56.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:56 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:45:56.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:56 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:56.786 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:56 smithi178 ceph-mon[47547]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-21T23:45:56.786 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:56 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:56.786 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:56 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:56.786 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:56 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:56.786 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:56 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:45:56.786 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:56 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:56.833 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:56 smithi176 ceph-mon[47172]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-21T23:45:56.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:56 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:56.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:56 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:56.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:56 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:45:56.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:56 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:45:56.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:56 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:56.872 INFO:teuthology.orchestra.run.smithi178.stdout:Created osd(s) 11 on host 'smithi178' 2024-02-21T23:45:56.872 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:56.887+0000 7f94e77fe700 1 -- 172.21.15.178:0/3378985150 <== mgr.14150 v2:172.21.15.90:6800/3697394685 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f94f819f4f0 con 0x7f94e00a1120 2024-02-21T23:45:56.875 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:56.890+0000 7f94fcb16700 1 -- 172.21.15.178:0/3378985150 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f94e00a1120 msgr2=0x7f94e00a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:45:56.875 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:56.890+0000 7f94fcb16700 1 --2- 172.21.15.178:0/3378985150 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f94e00a1120 0x7f94e00a35e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f94ec005950 tx=0x7f94ec005ed0 comp rx=0 tx=0).stop 2024-02-21T23:45:56.875 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:56.890+0000 7f94fcb16700 1 -- 172.21.15.178:0/3378985150 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f94f8101430 msgr2=0x7f94f819e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:45:56.875 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:56.890+0000 7f94fcb16700 1 --2- 172.21.15.178:0/3378985150 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f94f8101430 0x7f94f819e330 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f94f000b970 tx=0x7f94f000bc80 comp rx=0 tx=0).stop 2024-02-21T23:45:56.875 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:56.890+0000 7f94fcb16700 1 -- 172.21.15.178:0/3378985150 shutdown_connections 2024-02-21T23:45:56.875 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:56.890+0000 7f94fcb16700 1 --2- 172.21.15.178:0/3378985150 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f94e00a1120 0x7f94e00a35e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:56.875 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:56.890+0000 7f94fcb16700 1 --2- 172.21.15.178:0/3378985150 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f94f8101430 0x7f94f819e330 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:56.875 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:56.890+0000 7f94fcb16700 1 --2- 172.21.15.178:0/3378985150 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94f80fff00 0x7f94f8196e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:56.875 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:56.890+0000 7f94fcb16700 1 --2- 172.21.15.178:0/3378985150 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f94f8100910 0x7f94f8197340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:56.875 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:56.890+0000 7f94fcb16700 1 -- 172.21.15.178:0/3378985150 >> 172.21.15.178:0/3378985150 conn(0x7f94f80fa9d0 msgr2=0x7f94f80fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:45:56.876 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:56.890+0000 7f94fcb16700 1 -- 172.21.15.178:0/3378985150 shutdown_connections 2024-02-21T23:45:56.876 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:45:56.890+0000 7f94fcb16700 1 -- 172.21.15.178:0/3378985150 wait complete. 2024-02-21T23:45:57.389 DEBUG:teuthology.orchestra.run.smithi178:osd.11> sudo journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@osd.11.service 2024-02-21T23:45:57.392 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-02-21T23:45:57.392 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph osd stat -f json 2024-02-21T23:45:57.422 INFO:journalctl@ceph.osd.11.smithi178.stdout:-- Logs begin at Wed 2024-02-21 23:24:26 UTC. -- 2024-02-21T23:45:57.627 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.a/config 2024-02-21T23:45:57.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:57 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:57.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:57 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:57.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:57 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:45:57.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:57 smithi178 ceph-mon[47547]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-02-21T23:45:58.070 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:57 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:58.071 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:57 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:58.071 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:57 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:45:58.071 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:57 smithi090 ceph-mon[43385]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-02-21T23:45:58.083 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:57 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:58.083 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:57 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:45:58.083 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:57 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:45:58.083 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:57 smithi176 ceph-mon[47172]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-02-21T23:45:58.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.662+0000 7fe470fc2700 1 -- 172.21.15.90:0/2341523517 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe46c0ffc90 msgr2=0x7fe46c1000b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:45:58.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.662+0000 7fe470fc2700 1 --2- 172.21.15.90:0/2341523517 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe46c0ffc90 0x7fe46c1000b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe45c0099c0 tx=0x7fe45c009cd0 comp rx=0 tx=0).stop 2024-02-21T23:45:58.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.662+0000 7fe470fc2700 1 -- 172.21.15.90:0/2341523517 shutdown_connections 2024-02-21T23:45:58.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.662+0000 7fe470fc2700 1 --2- 172.21.15.90:0/2341523517 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe46c101830 0x7fe46c1080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:58.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.662+0000 7fe470fc2700 1 --2- 172.21.15.90:0/2341523517 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe46c0ffc90 0x7fe46c1000b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:58.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.662+0000 7fe470fc2700 1 --2- 172.21.15.90:0/2341523517 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe46c100e90 0x7fe46c1012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:58.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.662+0000 7fe470fc2700 1 -- 172.21.15.90:0/2341523517 >> 172.21.15.90:0/2341523517 conn(0x7fe46c0fb230 msgr2=0x7fe46c0fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:45:58.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.662+0000 7fe470fc2700 1 -- 172.21.15.90:0/2341523517 shutdown_connections 2024-02-21T23:45:58.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.662+0000 7fe470fc2700 1 -- 172.21.15.90:0/2341523517 wait complete. 2024-02-21T23:45:58.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.663+0000 7fe470fc2700 1 Processor -- start 2024-02-21T23:45:58.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.663+0000 7fe470fc2700 1 -- start start 2024-02-21T23:45:58.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.663+0000 7fe470fc2700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe46c0ffc90 0x7fe46c196ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:45:58.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.663+0000 7fe470fc2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe46c100e90 0x7fe46c197220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:45:58.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.663+0000 7fe470fc2700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe46c101830 0x7fe46c19e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:45:58.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.663+0000 7fe470fc2700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe46c19e810 con 0x7fe46c100e90 2024-02-21T23:45:58.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.663+0000 7fe470fc2700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe46c19e950 con 0x7fe46c0ffc90 2024-02-21T23:45:58.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.663+0000 7fe470fc2700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe46c19eb10 con 0x7fe46c101830 2024-02-21T23:45:58.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.663+0000 7fe469d9b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe46c100e90 0x7fe46c197220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:45:58.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.664+0000 7fe469d9b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe46c100e90 0x7fe46c197220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45640/0 (socket says 172.21.15.90:45640) 2024-02-21T23:45:58.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.664+0000 7fe469d9b700 1 -- 172.21.15.90:0/4149348757 learned_addr learned my addr 172.21.15.90:0/4149348757 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:45:58.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.664+0000 7fe46ad9d700 1 --2- 172.21.15.90:0/4149348757 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe46c101830 0x7fe46c19e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:45:58.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.664+0000 7fe46a59c700 1 --2- 172.21.15.90:0/4149348757 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe46c0ffc90 0x7fe46c196ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:45:58.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.664+0000 7fe469d9b700 1 -- 172.21.15.90:0/4149348757 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe46c101830 msgr2=0x7fe46c19e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:45:58.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.664+0000 7fe469d9b700 1 --2- 172.21.15.90:0/4149348757 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe46c101830 0x7fe46c19e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:58.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.664+0000 7fe469d9b700 1 -- 172.21.15.90:0/4149348757 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe46c0ffc90 msgr2=0x7fe46c196ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:45:58.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.664+0000 7fe469d9b700 1 --2- 172.21.15.90:0/4149348757 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe46c0ffc90 0x7fe46c196ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:58.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.664+0000 7fe469d9b700 1 -- 172.21.15.90:0/4149348757 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe45c0095f0 con 0x7fe46c100e90 2024-02-21T23:45:58.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.664+0000 7fe469d9b700 1 --2- 172.21.15.90:0/4149348757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe46c100e90 0x7fe46c197220 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe45400d980 tx=0x7fe45400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:45:58.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.664+0000 7fe4637fe700 1 -- 172.21.15.90:0/4149348757 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe454004220 con 0x7fe46c100e90 2024-02-21T23:45:58.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.664+0000 7fe4637fe700 1 -- 172.21.15.90:0/4149348757 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe454007640 con 0x7fe46c100e90 2024-02-21T23:45:58.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.664+0000 7fe470fc2700 1 -- 172.21.15.90:0/4149348757 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe46c19edf0 con 0x7fe46c100e90 2024-02-21T23:45:58.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.664+0000 7fe4637fe700 1 -- 172.21.15.90:0/4149348757 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe454005610 con 0x7fe46c100e90 2024-02-21T23:45:58.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.664+0000 7fe470fc2700 1 -- 172.21.15.90:0/4149348757 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe46c19f260 con 0x7fe46c100e90 2024-02-21T23:45:58.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.665+0000 7fe470fc2700 1 -- 172.21.15.90:0/4149348757 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe46c04ea40 con 0x7fe46c100e90 2024-02-21T23:45:58.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.666+0000 7fe4637fe700 1 -- 172.21.15.90:0/4149348757 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7fe454004750 con 0x7fe46c100e90 2024-02-21T23:45:58.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.667+0000 7fe4637fe700 1 --2- 172.21.15.90:0/4149348757 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fe4580a1120 0x7fe4580a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:45:58.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.667+0000 7fe46a59c700 1 --2- 172.21.15.90:0/4149348757 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fe4580a1120 0x7fe4580a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:45:58.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.667+0000 7fe4637fe700 1 -- 172.21.15.90:0/4149348757 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fe4540cc6a0 con 0x7fe46c100e90 2024-02-21T23:45:58.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.667+0000 7fe46a59c700 1 --2- 172.21.15.90:0/4149348757 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fe4580a1120 0x7fe4580a35e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe45c009990 tx=0x7fe45c01a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:45:58.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.670+0000 7fe4637fe700 1 -- 172.21.15.90:0/4149348757 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fe454096410 con 0x7fe46c100e90 2024-02-21T23:45:58.674 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:45:58 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:45:58.435+0000 7f3f0e7ca640 -1 osd.11 0 log_to_monitors true 2024-02-21T23:45:58.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.868+0000 7fe470fc2700 1 -- 172.21.15.90:0/4149348757 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fe46c19f540 con 0x7fe46c100e90 2024-02-21T23:45:58.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.868+0000 7fe4637fe700 1 -- 172.21.15.90:0/4149348757 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v74) v1 ==== 74+0+133 (secure 0 0 0) 0x7fe454099a30 con 0x7fe46c100e90 2024-02-21T23:45:58.852 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:45:58.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.870+0000 7fe470fc2700 1 -- 172.21.15.90:0/4149348757 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fe4580a1120 msgr2=0x7fe4580a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:45:58.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.870+0000 7fe470fc2700 1 --2- 172.21.15.90:0/4149348757 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fe4580a1120 0x7fe4580a35e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe45c009990 tx=0x7fe45c01a040 comp rx=0 tx=0).stop 2024-02-21T23:45:58.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.870+0000 7fe470fc2700 1 -- 172.21.15.90:0/4149348757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe46c100e90 msgr2=0x7fe46c197220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:45:58.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.870+0000 7fe470fc2700 1 --2- 172.21.15.90:0/4149348757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe46c100e90 0x7fe46c197220 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe45400d980 tx=0x7fe45400dc90 comp rx=0 tx=0).stop 2024-02-21T23:45:58.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.870+0000 7fe470fc2700 1 -- 172.21.15.90:0/4149348757 shutdown_connections 2024-02-21T23:45:58.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.870+0000 7fe470fc2700 1 --2- 172.21.15.90:0/4149348757 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fe4580a1120 0x7fe4580a35e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:58.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.870+0000 7fe470fc2700 1 --2- 172.21.15.90:0/4149348757 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe46c101830 0x7fe46c19e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:58.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.870+0000 7fe470fc2700 1 --2- 172.21.15.90:0/4149348757 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe46c0ffc90 0x7fe46c196ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:58.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.870+0000 7fe470fc2700 1 --2- 172.21.15.90:0/4149348757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe46c100e90 0x7fe46c197220 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:45:58.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.870+0000 7fe470fc2700 1 -- 172.21.15.90:0/4149348757 >> 172.21.15.90:0/4149348757 conn(0x7fe46c0fb230 msgr2=0x7fe46c068710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:45:58.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.871+0000 7fe470fc2700 1 -- 172.21.15.90:0/4149348757 shutdown_connections 2024-02-21T23:45:58.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:45:58.871+0000 7fe470fc2700 1 -- 172.21.15.90:0/4149348757 wait complete. 2024-02-21T23:45:59.083 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:58 smithi176 ceph-mon[47172]: from='osd.11 [v2:172.21.15.178:6824/4144536256,v1:172.21.15.178:6825/4144536256]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-02-21T23:45:59.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:58 smithi176 ceph-mon[47172]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-02-21T23:45:59.120 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:58 smithi178 ceph-mon[47547]: from='osd.11 [v2:172.21.15.178:6824/4144536256,v1:172.21.15.178:6825/4144536256]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-02-21T23:45:59.120 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:58 smithi178 ceph-mon[47547]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-02-21T23:45:59.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:58 smithi090 ceph-mon[43385]: from='osd.11 [v2:172.21.15.178:6824/4144536256,v1:172.21.15.178:6825/4144536256]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-02-21T23:45:59.191 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:58 smithi090 ceph-mon[43385]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-02-21T23:45:59.275 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":74,"num_osds":12,"num_up_osds":11,"osd_up_since":1708559141,"num_in_osds":12,"osd_in_since":1708559144,"num_remapped_pgs":0} 2024-02-21T23:46:00.174 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:45:59 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:45:59.866+0000 7f3f03e2e700 -1 osd.11 0 waiting for initial osdmap 2024-02-21T23:46:00.174 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:45:59 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:45:59.880+0000 7f3efdc1f700 -1 osd.11 75 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-21T23:46:00.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:59 smithi178 ceph-mon[47547]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-02-21T23:46:00.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:59 smithi178 ceph-mon[47547]: osdmap e74: 12 total, 11 up, 12 in 2024-02-21T23:46:00.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:59 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-21T23:46:00.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:59 smithi178 ceph-mon[47547]: from='osd.11 [v2:172.21.15.178:6824/4144536256,v1:172.21.15.178:6825/4144536256]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-02-21T23:46:00.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:59 smithi178 ceph-mon[47547]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-02-21T23:46:00.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:59 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/4149348757' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-21T23:46:00.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:59 smithi178 ceph-mon[47547]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-02-21T23:46:00.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:59 smithi178 ceph-mon[47547]: Detected new or changed devices on smithi178 2024-02-21T23:46:00.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:59 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:00.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:59 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:00.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:59 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:46:00.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:59 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:46:00.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:59 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:46:00.176 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:59 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:46:00.176 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:59 smithi178 ceph-mon[47547]: Adjusting osd_memory_target on smithi178 to 4278M 2024-02-21T23:46:00.176 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:59 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:00.176 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:59 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:00.176 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:59 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:46:00.176 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:45:59 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:00.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:59 smithi090 ceph-mon[43385]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-02-21T23:46:00.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:59 smithi090 ceph-mon[43385]: osdmap e74: 12 total, 11 up, 12 in 2024-02-21T23:46:00.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:59 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-21T23:46:00.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:59 smithi090 ceph-mon[43385]: from='osd.11 [v2:172.21.15.178:6824/4144536256,v1:172.21.15.178:6825/4144536256]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-02-21T23:46:00.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:59 smithi090 ceph-mon[43385]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-02-21T23:46:00.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:59 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/4149348757' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-21T23:46:00.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:59 smithi090 ceph-mon[43385]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-02-21T23:46:00.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:59 smithi090 ceph-mon[43385]: Detected new or changed devices on smithi178 2024-02-21T23:46:00.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:59 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:00.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:59 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:00.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:59 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:46:00.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:59 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:46:00.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:59 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:46:00.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:59 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:46:00.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:59 smithi090 ceph-mon[43385]: Adjusting osd_memory_target on smithi178 to 4278M 2024-02-21T23:46:00.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:59 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:00.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:59 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:00.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:59 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:46:00.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:45:59 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:00.276 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph osd stat -f json 2024-02-21T23:46:00.333 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:59 smithi176 ceph-mon[47172]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-02-21T23:46:00.333 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:59 smithi176 ceph-mon[47172]: osdmap e74: 12 total, 11 up, 12 in 2024-02-21T23:46:00.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:59 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-21T23:46:00.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:59 smithi176 ceph-mon[47172]: from='osd.11 [v2:172.21.15.178:6824/4144536256,v1:172.21.15.178:6825/4144536256]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-02-21T23:46:00.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:59 smithi176 ceph-mon[47172]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-02-21T23:46:00.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:59 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/4149348757' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-21T23:46:00.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:59 smithi176 ceph-mon[47172]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-02-21T23:46:00.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:59 smithi176 ceph-mon[47172]: Detected new or changed devices on smithi178 2024-02-21T23:46:00.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:59 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:00.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:59 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:00.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:59 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:46:00.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:59 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:46:00.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:59 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:46:00.335 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:59 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-02-21T23:46:00.335 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:59 smithi176 ceph-mon[47172]: Adjusting osd_memory_target on smithi178 to 4278M 2024-02-21T23:46:00.335 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:59 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:00.335 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:59 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:00.335 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:59 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:46:00.335 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:45:59 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:00.498 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.a/config 2024-02-21T23:46:00.954 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:00 smithi090 ceph-mon[43385]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-02-21T23:46:00.954 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:00 smithi090 ceph-mon[43385]: osdmap e75: 12 total, 11 up, 12 in 2024-02-21T23:46:00.954 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:00 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-21T23:46:00.954 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:00 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-21T23:46:01.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:00 smithi178 ceph-mon[47547]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-02-21T23:46:01.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:00 smithi178 ceph-mon[47547]: osdmap e75: 12 total, 11 up, 12 in 2024-02-21T23:46:01.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:00 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-21T23:46:01.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:00 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-21T23:46:01.333 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:00 smithi176 ceph-mon[47172]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-02-21T23:46:01.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:00 smithi176 ceph-mon[47172]: osdmap e75: 12 total, 11 up, 12 in 2024-02-21T23:46:01.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:00 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-21T23:46:01.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:00 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-21T23:46:01.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.495+0000 7f5cb6bb5700 1 -- 172.21.15.90:0/3966634333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5cb00ffc90 msgr2=0x7f5cb01000b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:01.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.495+0000 7f5cb6bb5700 1 --2- 172.21.15.90:0/3966634333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5cb00ffc90 0x7f5cb01000b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5ca40099c0 tx=0x7f5ca4009cd0 comp rx=0 tx=0).stop 2024-02-21T23:46:01.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.495+0000 7f5cb6bb5700 1 -- 172.21.15.90:0/3966634333 shutdown_connections 2024-02-21T23:46:01.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.495+0000 7f5cb6bb5700 1 --2- 172.21.15.90:0/3966634333 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5cb0101830 0x7f5cb01080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:01.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.495+0000 7f5cb6bb5700 1 --2- 172.21.15.90:0/3966634333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5cb00ffc90 0x7f5cb01000b0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:01.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.495+0000 7f5cb6bb5700 1 --2- 172.21.15.90:0/3966634333 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5cb0100e90 0x7f5cb01012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:01.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.495+0000 7f5cb6bb5700 1 -- 172.21.15.90:0/3966634333 >> 172.21.15.90:0/3966634333 conn(0x7f5cb00fb230 msgr2=0x7f5cb00fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:01.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.495+0000 7f5cb6bb5700 1 -- 172.21.15.90:0/3966634333 shutdown_connections 2024-02-21T23:46:01.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.495+0000 7f5cb6bb5700 1 -- 172.21.15.90:0/3966634333 wait complete. 2024-02-21T23:46:01.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.496+0000 7f5cb6bb5700 1 Processor -- start 2024-02-21T23:46:01.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.496+0000 7f5cb6bb5700 1 -- start start 2024-02-21T23:46:01.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.496+0000 7f5cb6bb5700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5cb00ffc90 0x7f5cb010a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:01.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.496+0000 7f5cb6bb5700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5cb0100e90 0x7f5cb0108bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:01.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.496+0000 7f5cb6bb5700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5cb0101830 0x7f5cb0109130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:01.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.496+0000 7f5cb6bb5700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb01098d0 con 0x7f5cb0100e90 2024-02-21T23:46:01.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.496+0000 7f5cb6bb5700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb0109a40 con 0x7f5cb00ffc90 2024-02-21T23:46:01.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.496+0000 7f5cb6bb5700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb01a0cd0 con 0x7f5cb0101830 2024-02-21T23:46:01.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.496+0000 7f5caffff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5cb0100e90 0x7f5cb0108bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:01.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.496+0000 7f5caffff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5cb0100e90 0x7f5cb0108bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45664/0 (socket says 172.21.15.90:45664) 2024-02-21T23:46:01.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.496+0000 7f5caffff700 1 -- 172.21.15.90:0/1990950275 learned_addr learned my addr 172.21.15.90:0/1990950275 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:01.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.496+0000 7f5cb5152700 1 --2- 172.21.15.90:0/1990950275 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5cb0101830 0x7f5cb0109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:01.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.496+0000 7f5cb4951700 1 --2- 172.21.15.90:0/1990950275 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5cb00ffc90 0x7f5cb010a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:01.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.496+0000 7f5caffff700 1 -- 172.21.15.90:0/1990950275 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5cb0101830 msgr2=0x7f5cb0109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:01.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.496+0000 7f5caffff700 1 --2- 172.21.15.90:0/1990950275 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5cb0101830 0x7f5cb0109130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:01.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.496+0000 7f5caffff700 1 -- 172.21.15.90:0/1990950275 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5cb00ffc90 msgr2=0x7f5cb010a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:01.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.496+0000 7f5caffff700 1 --2- 172.21.15.90:0/1990950275 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5cb00ffc90 0x7f5cb010a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:01.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.496+0000 7f5caffff700 1 -- 172.21.15.90:0/1990950275 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca40095f0 con 0x7f5cb0100e90 2024-02-21T23:46:01.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.497+0000 7f5caffff700 1 --2- 172.21.15.90:0/1990950275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5cb0100e90 0x7f5cb0108bf0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5c9c00d950 tx=0x7f5c9c00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:01.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.497+0000 7f5cadffb700 1 -- 172.21.15.90:0/1990950275 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c9c009b40 con 0x7f5cb0100e90 2024-02-21T23:46:01.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.497+0000 7f5cadffb700 1 -- 172.21.15.90:0/1990950275 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5c9c009ca0 con 0x7f5cb0100e90 2024-02-21T23:46:01.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.497+0000 7f5cadffb700 1 -- 172.21.15.90:0/1990950275 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c9c00f620 con 0x7f5cb0100e90 2024-02-21T23:46:01.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.497+0000 7f5cb6bb5700 1 -- 172.21.15.90:0/1990950275 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cb01a0f60 con 0x7f5cb0100e90 2024-02-21T23:46:01.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.497+0000 7f5cb6bb5700 1 -- 172.21.15.90:0/1990950275 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cb01a1370 con 0x7f5cb0100e90 2024-02-21T23:46:01.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.498+0000 7f5cb6bb5700 1 -- 172.21.15.90:0/1990950275 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cb01070f0 con 0x7f5cb0100e90 2024-02-21T23:46:01.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.499+0000 7f5cadffb700 1 -- 172.21.15.90:0/1990950275 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7f5c9c010460 con 0x7f5cb0100e90 2024-02-21T23:46:01.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.500+0000 7f5cadffb700 1 --2- 172.21.15.90:0/1990950275 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f5ca00a1120 0x7f5ca00a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:01.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.500+0000 7f5cb4951700 1 --2- 172.21.15.90:0/1990950275 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f5ca00a1120 0x7f5ca00a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:01.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.500+0000 7f5cadffb700 1 -- 172.21.15.90:0/1990950275 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f5c9c0cc7c0 con 0x7f5cb0100e90 2024-02-21T23:46:01.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.500+0000 7f5cb4951700 1 --2- 172.21.15.90:0/1990950275 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f5ca00a1120 0x7f5ca00a35e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5ca400b580 tx=0x7f5ca4005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:01.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.504+0000 7f5cadffb700 1 -- 172.21.15.90:0/1990950275 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f5c9c0963f0 con 0x7f5cb0100e90 2024-02-21T23:46:01.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.699+0000 7f5cb6bb5700 1 -- 172.21.15.90:0/1990950275 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f5cb002ce00 con 0x7f5cb0100e90 2024-02-21T23:46:01.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.699+0000 7f5cadffb700 1 -- 172.21.15.90:0/1990950275 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v76) v1 ==== 74+0+133 (secure 0 0 0) 0x7f5c9c099a10 con 0x7f5cb0100e90 2024-02-21T23:46:01.683 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:46:01.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.701+0000 7f5cb6bb5700 1 -- 172.21.15.90:0/1990950275 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f5ca00a1120 msgr2=0x7f5ca00a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:01.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.701+0000 7f5cb6bb5700 1 --2- 172.21.15.90:0/1990950275 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f5ca00a1120 0x7f5ca00a35e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5ca400b580 tx=0x7f5ca4005fb0 comp rx=0 tx=0).stop 2024-02-21T23:46:01.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.701+0000 7f5cb6bb5700 1 -- 172.21.15.90:0/1990950275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5cb0100e90 msgr2=0x7f5cb0108bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:01.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.701+0000 7f5cb6bb5700 1 --2- 172.21.15.90:0/1990950275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5cb0100e90 0x7f5cb0108bf0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5c9c00d950 tx=0x7f5c9c00dd10 comp rx=0 tx=0).stop 2024-02-21T23:46:01.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.702+0000 7f5cb6bb5700 1 -- 172.21.15.90:0/1990950275 shutdown_connections 2024-02-21T23:46:01.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.702+0000 7f5cb6bb5700 1 --2- 172.21.15.90:0/1990950275 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f5ca00a1120 0x7f5ca00a35e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:01.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.702+0000 7f5cb6bb5700 1 --2- 172.21.15.90:0/1990950275 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5cb0101830 0x7f5cb0109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:01.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.702+0000 7f5cb6bb5700 1 --2- 172.21.15.90:0/1990950275 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5cb00ffc90 0x7f5cb010a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:01.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.702+0000 7f5cb6bb5700 1 --2- 172.21.15.90:0/1990950275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5cb0100e90 0x7f5cb0108bf0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:01.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.702+0000 7f5cb6bb5700 1 -- 172.21.15.90:0/1990950275 >> 172.21.15.90:0/1990950275 conn(0x7f5cb00fb230 msgr2=0x7f5cb0104230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:01.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.702+0000 7f5cb6bb5700 1 -- 172.21.15.90:0/1990950275 shutdown_connections 2024-02-21T23:46:01.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:01.702+0000 7f5cb6bb5700 1 -- 172.21.15.90:0/1990950275 wait complete. 2024-02-21T23:46:02.058 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":76,"num_osds":12,"num_up_osds":12,"osd_up_since":1708559160,"num_in_osds":12,"osd_in_since":1708559144,"num_remapped_pgs":0} 2024-02-21T23:46:02.059 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph osd dump --format=json 2024-02-21T23:46:02.173 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:01 smithi178 ceph-mon[47547]: purged_snaps scrub starts 2024-02-21T23:46:02.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:01 smithi178 ceph-mon[47547]: purged_snaps scrub ok 2024-02-21T23:46:02.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:01 smithi178 ceph-mon[47547]: osd.11 [v2:172.21.15.178:6824/4144536256,v1:172.21.15.178:6825/4144536256] boot 2024-02-21T23:46:02.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:01 smithi178 ceph-mon[47547]: osdmap e76: 12 total, 12 up, 12 in 2024-02-21T23:46:02.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:01 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-21T23:46:02.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:01 smithi178 ceph-mon[47547]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:02.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:01 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/1990950275' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-21T23:46:02.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:01 smithi090 ceph-mon[43385]: purged_snaps scrub starts 2024-02-21T23:46:02.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:01 smithi090 ceph-mon[43385]: purged_snaps scrub ok 2024-02-21T23:46:02.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:01 smithi090 ceph-mon[43385]: osd.11 [v2:172.21.15.178:6824/4144536256,v1:172.21.15.178:6825/4144536256] boot 2024-02-21T23:46:02.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:01 smithi090 ceph-mon[43385]: osdmap e76: 12 total, 12 up, 12 in 2024-02-21T23:46:02.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:01 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-21T23:46:02.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:01 smithi090 ceph-mon[43385]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:02.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:01 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/1990950275' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-21T23:46:02.277 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.a/config 2024-02-21T23:46:02.333 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:01 smithi176 ceph-mon[47172]: purged_snaps scrub starts 2024-02-21T23:46:02.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:01 smithi176 ceph-mon[47172]: purged_snaps scrub ok 2024-02-21T23:46:02.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:01 smithi176 ceph-mon[47172]: osd.11 [v2:172.21.15.178:6824/4144536256,v1:172.21.15.178:6825/4144536256] boot 2024-02-21T23:46:02.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:01 smithi176 ceph-mon[47172]: osdmap e76: 12 total, 12 up, 12 in 2024-02-21T23:46:02.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:01 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-21T23:46:02.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:01 smithi176 ceph-mon[47172]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:02.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:01 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/1990950275' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-21T23:46:02.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:02 smithi090 ceph-mon[43385]: osdmap e77: 12 total, 12 up, 12 in 2024-02-21T23:46:03.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:02 smithi178 ceph-mon[47547]: osdmap e77: 12 total, 12 up, 12 in 2024-02-21T23:46:03.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.245+0000 7fe1643f5700 1 -- 172.21.15.90:0/139614836 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe15c069100 msgr2=0x7fe15c10db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:03.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.245+0000 7fe1643f5700 1 --2- 172.21.15.90:0/139614836 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe15c069100 0x7fe15c10db30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe15400b0a0 tx=0x7fe15400b3b0 comp rx=0 tx=0).stop 2024-02-21T23:46:03.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.246+0000 7fe1643f5700 1 -- 172.21.15.90:0/139614836 shutdown_connections 2024-02-21T23:46:03.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.246+0000 7fe1643f5700 1 --2- 172.21.15.90:0/139614836 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe15c069100 0x7fe15c10db30 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:03.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.246+0000 7fe1643f5700 1 --2- 172.21.15.90:0/139614836 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe15c106e80 0x7fe15c107260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:03.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.246+0000 7fe1643f5700 1 --2- 172.21.15.90:0/139614836 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe15c068740 0x7fe15c068bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:03.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.246+0000 7fe1643f5700 1 -- 172.21.15.90:0/139614836 >> 172.21.15.90:0/139614836 conn(0x7fe15c076d70 msgr2=0x7fe15c077180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:03.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.246+0000 7fe1643f5700 1 -- 172.21.15.90:0/139614836 shutdown_connections 2024-02-21T23:46:03.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.246+0000 7fe1643f5700 1 -- 172.21.15.90:0/139614836 wait complete. 2024-02-21T23:46:03.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.246+0000 7fe1643f5700 1 Processor -- start 2024-02-21T23:46:03.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.247+0000 7fe1643f5700 1 -- start start 2024-02-21T23:46:03.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.247+0000 7fe1643f5700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe15c068740 0x7fe15c197160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:03.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.247+0000 7fe1643f5700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe15c069100 0x7fe15c1976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:03.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.247+0000 7fe1643f5700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe15c106e80 0x7fe15c19ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:03.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.247+0000 7fe1643f5700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c19c060 con 0x7fe15c068740 2024-02-21T23:46:03.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.247+0000 7fe1643f5700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c19c1d0 con 0x7fe15c069100 2024-02-21T23:46:03.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.247+0000 7fe1643f5700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c19c390 con 0x7fe15c106e80 2024-02-21T23:46:03.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.247+0000 7fe162191700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe15c068740 0x7fe15c197160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:03.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.247+0000 7fe162191700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe15c068740 0x7fe15c197160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45680/0 (socket says 172.21.15.90:45680) 2024-02-21T23:46:03.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.247+0000 7fe162191700 1 -- 172.21.15.90:0/2826340897 learned_addr learned my addr 172.21.15.90:0/2826340897 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:03.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.247+0000 7fe162992700 1 --2- 172.21.15.90:0/2826340897 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe15c106e80 0x7fe15c19ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:03.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.247+0000 7fe161990700 1 --2- 172.21.15.90:0/2826340897 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe15c069100 0x7fe15c1976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:03.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.247+0000 7fe162191700 1 -- 172.21.15.90:0/2826340897 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe15c106e80 msgr2=0x7fe15c19ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:03.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.247+0000 7fe162191700 1 --2- 172.21.15.90:0/2826340897 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe15c106e80 0x7fe15c19ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:03.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.247+0000 7fe162191700 1 -- 172.21.15.90:0/2826340897 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe15c069100 msgr2=0x7fe15c1976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:03.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.247+0000 7fe162191700 1 --2- 172.21.15.90:0/2826340897 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe15c069100 0x7fe15c1976a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:03.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.247+0000 7fe162191700 1 -- 172.21.15.90:0/2826340897 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe154009d00 con 0x7fe15c068740 2024-02-21T23:46:03.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.247+0000 7fe162191700 1 --2- 172.21.15.90:0/2826340897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe15c068740 0x7fe15c197160 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe15800c920 tx=0x7fe15800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:03.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.248+0000 7fe1537fe700 1 -- 172.21.15.90:0/2826340897 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe158007b00 con 0x7fe15c068740 2024-02-21T23:46:03.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.248+0000 7fe1537fe700 1 -- 172.21.15.90:0/2826340897 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe15800ce70 con 0x7fe15c068740 2024-02-21T23:46:03.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.248+0000 7fe1643f5700 1 -- 172.21.15.90:0/2826340897 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe15c19c670 con 0x7fe15c068740 2024-02-21T23:46:03.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.248+0000 7fe1537fe700 1 -- 172.21.15.90:0/2826340897 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1580054e0 con 0x7fe15c068740 2024-02-21T23:46:03.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.248+0000 7fe1643f5700 1 -- 172.21.15.90:0/2826340897 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe15c1a3ed0 con 0x7fe15c068740 2024-02-21T23:46:03.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.249+0000 7fe1643f5700 1 -- 172.21.15.90:0/2826340897 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe15c109210 con 0x7fe15c068740 2024-02-21T23:46:03.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.250+0000 7fe1537fe700 1 -- 172.21.15.90:0/2826340897 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7fe158004750 con 0x7fe15c068740 2024-02-21T23:46:03.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.250+0000 7fe1537fe700 1 --2- 172.21.15.90:0/2826340897 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fe1480a1220 0x7fe1480a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:03.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.250+0000 7fe161990700 1 --2- 172.21.15.90:0/2826340897 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fe1480a1220 0x7fe1480a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:03.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.251+0000 7fe1537fe700 1 -- 172.21.15.90:0/2826340897 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe1580cbfc0 con 0x7fe15c068740 2024-02-21T23:46:03.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.251+0000 7fe161990700 1 --2- 172.21.15.90:0/2826340897 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fe1480a1220 0x7fe1480a36e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe14c005fd0 tx=0x7fe14c00a560 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:03.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.255+0000 7fe1537fe700 1 -- 172.21.15.90:0/2826340897 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fe158095b50 con 0x7fe15c068740 2024-02-21T23:46:03.333 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:02 smithi176 ceph-mon[47172]: osdmap e77: 12 total, 12 up, 12 in 2024-02-21T23:46:03.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.451+0000 7fe1643f5700 1 -- 172.21.15.90:0/2826340897 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe15c198470 con 0x7fe15c068740 2024-02-21T23:46:03.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.452+0000 7fe1537fe700 1 -- 172.21.15.90:0/2826340897 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+18387 (secure 0 0 0) 0x7fe158099170 con 0x7fe15c068740 2024-02-21T23:46:03.435 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:46:03.435 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":77,"fsid":"7ed81218-d112-11ee-95bd-87774f69a715","created":"2024-02-21T23:40:41.785526+0000","modified":"2024-02-21T23:46:01.867321+0000","last_up_change":"2024-02-21T23:46:00.860501+0000","last_in_change":"2024-02-21T23:45:44.423815+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:43:21.675789+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"77878890-d9a5-4d5b-a943-fec29ee0446e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6803","nonce":885319826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6805","nonce":885319826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6809","nonce":885319826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6807","nonce":885319826}]},"public_addr":"172.21.15.90:6803/885319826","cluster_addr":"172.21.15.90:6805/885319826","heartbeat_back_addr":"172.21.15.90:6809/885319826","heartbeat_front_addr":"172.21.15.90:6807/885319826","state":["exists","up"]},{"osd":1,"uuid":"7d6ea950-7a57-4f13-ab38-10a658912783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6801","nonce":4085240925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6803","nonce":4085240925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6807","nonce":4085240925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6805","nonce":4085240925}]},"public_addr":"172.21.15.176:6801/4085240925","cluster_addr":"172.21.15.176:6803/4085240925","heartbeat_back_addr":"172.21.15.176:6807/4085240925","heartbeat_front_addr":"172.21.15.176:6805/4085240925","state":["exists","up"]},{"osd":2,"uuid":"b83486bb-27f2-4d56-97fb-fb336fec4b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6801","nonce":3597250789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6803","nonce":3597250789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6807","nonce":3597250789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6805","nonce":3597250789}]},"public_addr":"172.21.15.178:6801/3597250789","cluster_addr":"172.21.15.178:6803/3597250789","heartbeat_back_addr":"172.21.15.178:6807/3597250789","heartbeat_front_addr":"172.21.15.178:6805/3597250789","state":["exists","up"]},{"osd":3,"uuid":"870e5bb6-eed5-4a0a-8e03-38de4f816b44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6811","nonce":3299043821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6813","nonce":3299043821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6817","nonce":3299043821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6815","nonce":3299043821}]},"public_addr":"172.21.15.90:6811/3299043821","cluster_addr":"172.21.15.90:6813/3299043821","heartbeat_back_addr":"172.21.15.90:6817/3299043821","heartbeat_front_addr":"172.21.15.90:6815/3299043821","state":["exists","up"]},{"osd":4,"uuid":"cb9f7f48-76e0-4368-ab87-2a09a8487ccd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6809","nonce":3548915687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6811","nonce":3548915687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6815","nonce":3548915687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6813","nonce":3548915687}]},"public_addr":"172.21.15.176:6809/3548915687","cluster_addr":"172.21.15.176:6811/3548915687","heartbeat_back_addr":"172.21.15.176:6815/3548915687","heartbeat_front_addr":"172.21.15.176:6813/3548915687","state":["exists","up"]},{"osd":5,"uuid":"54749022-c50d-4831-a1d6-8412b59c8a01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6809","nonce":286274518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6811","nonce":286274518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6815","nonce":286274518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6813","nonce":286274518}]},"public_addr":"172.21.15.178:6809/286274518","cluster_addr":"172.21.15.178:6811/286274518","heartbeat_back_addr":"172.21.15.178:6815/286274518","heartbeat_front_addr":"172.21.15.178:6813/286274518","state":["exists","up"]},{"osd":6,"uuid":"09bb5393-2efd-4a63-b0e8-99bef5bced13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6819","nonce":67640620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6821","nonce":67640620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6825","nonce":67640620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6823","nonce":67640620}]},"public_addr":"172.21.15.90:6819/67640620","cluster_addr":"172.21.15.90:6821/67640620","heartbeat_back_addr":"172.21.15.90:6825/67640620","heartbeat_front_addr":"172.21.15.90:6823/67640620","state":["exists","up"]},{"osd":7,"uuid":"7a859e69-6b63-467b-a3b3-99247d66f5c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6817","nonce":3998561598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6819","nonce":3998561598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6823","nonce":3998561598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6821","nonce":3998561598}]},"public_addr":"172.21.15.176:6817/3998561598","cluster_addr":"172.21.15.176:6819/3998561598","heartbeat_back_addr":"172.21.15.176:6823/3998561598","heartbeat_front_addr":"172.21.15.176:6821/3998561598","state":["exists","up"]},{"osd":8,"uuid":"bc23144e-22a5-4894-a8b5-9323383574d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6817","nonce":490229963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6819","nonce":490229963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6823","nonce":490229963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6821","nonce":490229963}]},"public_addr":"172.21.15.178:6817/490229963","cluster_addr":"172.21.15.178:6819/490229963","heartbeat_back_addr":"172.21.15.178:6823/490229963","heartbeat_front_addr":"172.21.15.178:6821/490229963","state":["exists","up"]},{"osd":9,"uuid":"0c2cecd4-55de-4fc4-a6f4-ca97c8460444","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6827","nonce":2061394761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6829","nonce":2061394761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6833","nonce":2061394761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6831","nonce":2061394761}]},"public_addr":"172.21.15.90:6827/2061394761","cluster_addr":"172.21.15.90:6829/2061394761","heartbeat_back_addr":"172.21.15.90:6833/2061394761","heartbeat_front_addr":"172.21.15.90:6831/2061394761","state":["exists","up"]},{"osd":10,"uuid":"fa04232b-b1ec-447a-8e94-7c9bf26d0291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6825","nonce":639411757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6827","nonce":639411757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6831","nonce":639411757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6829","nonce":639411757}]},"public_addr":"172.21.15.176:6825/639411757","cluster_addr":"172.21.15.176:6827/639411757","heartbeat_back_addr":"172.21.15.176:6831/639411757","heartbeat_front_addr":"172.21.15.176:6829/639411757","state":["exists","up"]},{"osd":11,"uuid":"7282e294-64fc-4856-9c74-86115f69408c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6825","nonce":4144536256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6827","nonce":4144536256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6831","nonce":4144536256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6829","nonce":4144536256}]},"public_addr":"172.21.15.178:6825/4144536256","cluster_addr":"172.21.15.178:6827/4144536256","heartbeat_back_addr":"172.21.15.178:6831/4144536256","heartbeat_front_addr":"172.21.15.178:6829/4144536256","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:42:44.280345+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:01.235630+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:19.408896+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:36.201749+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:53.401854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:44:11.342158+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:44:29.324502+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:44:46.821595+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:04.642344+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:23.019004+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:41.140149+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:59.401951+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/159655312":"2024-02-22T23:41:33.360021+0000","172.21.15.90:6801/1385551604":"2024-02-22T23:41:33.360021+0000","172.21.15.90:0/1452574770":"2024-02-22T23:41:33.360021+0000","172.21.15.90:0/3815292343":"2024-02-22T23:41:11.484145+0000","172.21.15.90:6800/1385551604":"2024-02-22T23:41:33.360021+0000","172.21.15.90:6801/4247695574":"2024-02-22T23:41:11.484145+0000","172.21.15.90:0/1328266622":"2024-02-22T23:41:11.484145+0000","172.21.15.90:6800/4247695574":"2024-02-22T23:41:11.484145+0000","172.21.15.90:0/2569421522":"2024-02-22T23:41:11.484145+0000","172.21.15.90:0/3019701027":"2024-02-22T23:41:33.360021+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:46:03.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.454+0000 7fe1643f5700 1 -- 172.21.15.90:0/2826340897 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fe1480a1220 msgr2=0x7fe1480a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:03.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.454+0000 7fe1643f5700 1 --2- 172.21.15.90:0/2826340897 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fe1480a1220 0x7fe1480a36e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe14c005fd0 tx=0x7fe14c00a560 comp rx=0 tx=0).stop 2024-02-21T23:46:03.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.454+0000 7fe1643f5700 1 -- 172.21.15.90:0/2826340897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe15c068740 msgr2=0x7fe15c197160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:03.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.454+0000 7fe1643f5700 1 --2- 172.21.15.90:0/2826340897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe15c068740 0x7fe15c197160 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe15800c920 tx=0x7fe15800cc30 comp rx=0 tx=0).stop 2024-02-21T23:46:03.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.454+0000 7fe1643f5700 1 -- 172.21.15.90:0/2826340897 shutdown_connections 2024-02-21T23:46:03.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.454+0000 7fe1643f5700 1 --2- 172.21.15.90:0/2826340897 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fe1480a1220 0x7fe1480a36e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:03.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.454+0000 7fe1643f5700 1 --2- 172.21.15.90:0/2826340897 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe15c106e80 0x7fe15c19ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:03.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.454+0000 7fe1643f5700 1 --2- 172.21.15.90:0/2826340897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe15c068740 0x7fe15c197160 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:03.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.454+0000 7fe1643f5700 1 --2- 172.21.15.90:0/2826340897 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe15c069100 0x7fe15c1976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:03.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.454+0000 7fe1643f5700 1 -- 172.21.15.90:0/2826340897 >> 172.21.15.90:0/2826340897 conn(0x7fe15c076d70 msgr2=0x7fe15c0fe8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:03.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.454+0000 7fe1643f5700 1 -- 172.21.15.90:0/2826340897 shutdown_connections 2024-02-21T23:46:03.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:03.454+0000 7fe1643f5700 1 -- 172.21.15.90:0/2826340897 wait complete. 2024-02-21T23:46:03.824 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:43:21.675789+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:46:03.825 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-02-21T23:46:04.045 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.a/config 2024-02-21T23:46:04.144 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:03 smithi090 ceph-mon[43385]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:04.144 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:03 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/2826340897' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-21T23:46:04.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:03 smithi178 ceph-mon[47547]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:04.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:03 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/2826340897' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-21T23:46:04.333 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:03 smithi176 ceph-mon[47172]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:04.333 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:03 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/2826340897' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-21T23:46:04.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.949+0000 7f59f3922700 1 -- 172.21.15.90:0/489516652 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59ec069100 msgr2=0x7f59ec10db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:04.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.949+0000 7f59f3922700 1 --2- 172.21.15.90:0/489516652 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59ec069100 0x7f59ec10db30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f59e400b0a0 tx=0x7f59e400b3b0 comp rx=0 tx=0).stop 2024-02-21T23:46:04.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.949+0000 7f59f3922700 1 -- 172.21.15.90:0/489516652 shutdown_connections 2024-02-21T23:46:04.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.949+0000 7f59f3922700 1 --2- 172.21.15.90:0/489516652 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59ec069100 0x7f59ec10db30 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:04.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.949+0000 7f59f3922700 1 --2- 172.21.15.90:0/489516652 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59ec106e80 0x7f59ec107260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:04.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.949+0000 7f59f3922700 1 --2- 172.21.15.90:0/489516652 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59ec068740 0x7f59ec068bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:04.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.949+0000 7f59f3922700 1 -- 172.21.15.90:0/489516652 >> 172.21.15.90:0/489516652 conn(0x7f59ec076d70 msgr2=0x7f59ec077180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:04.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.949+0000 7f59f3922700 1 -- 172.21.15.90:0/489516652 shutdown_connections 2024-02-21T23:46:04.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.949+0000 7f59f3922700 1 -- 172.21.15.90:0/489516652 wait complete. 2024-02-21T23:46:04.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.950+0000 7f59f3922700 1 Processor -- start 2024-02-21T23:46:04.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.950+0000 7f59f3922700 1 -- start start 2024-02-21T23:46:04.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.950+0000 7f59f3922700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59ec068740 0x7f59ec0ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:04.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.950+0000 7f59f3922700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59ec069100 0x7f59ec1001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:04.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.950+0000 7f59f3922700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59ec106e80 0x7f59ec100720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:04.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.950+0000 7f59f3922700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59ec1a5500 con 0x7f59ec106e80 2024-02-21T23:46:04.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.950+0000 7f59f3922700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59ec1a5670 con 0x7f59ec069100 2024-02-21T23:46:04.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.950+0000 7f59f3922700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59ec1a5830 con 0x7f59ec068740 2024-02-21T23:46:04.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.950+0000 7f59f1ebf700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59ec106e80 0x7f59ec100720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:04.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.950+0000 7f59f16be700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59ec068740 0x7f59ec0ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:04.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.950+0000 7f59f1ebf700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59ec106e80 0x7f59ec100720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45698/0 (socket says 172.21.15.90:45698) 2024-02-21T23:46:04.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.950+0000 7f59f1ebf700 1 -- 172.21.15.90:0/1570722742 learned_addr learned my addr 172.21.15.90:0/1570722742 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:04.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.950+0000 7f59f0ebd700 1 --2- 172.21.15.90:0/1570722742 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59ec069100 0x7f59ec1001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:04.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.950+0000 7f59f1ebf700 1 -- 172.21.15.90:0/1570722742 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59ec068740 msgr2=0x7f59ec0ffca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:04.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.950+0000 7f59f1ebf700 1 --2- 172.21.15.90:0/1570722742 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59ec068740 0x7f59ec0ffca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:04.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.950+0000 7f59f1ebf700 1 -- 172.21.15.90:0/1570722742 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59ec069100 msgr2=0x7f59ec1001e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:04.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.950+0000 7f59f1ebf700 1 --2- 172.21.15.90:0/1570722742 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59ec069100 0x7f59ec1001e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:04.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.950+0000 7f59f1ebf700 1 -- 172.21.15.90:0/1570722742 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e4009d00 con 0x7f59ec106e80 2024-02-21T23:46:04.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.951+0000 7f59f1ebf700 1 --2- 172.21.15.90:0/1570722742 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59ec106e80 0x7f59ec100720 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f59e400b070 tx=0x7f59e40094c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:04.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.951+0000 7f59e27fc700 1 -- 172.21.15.90:0/1570722742 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e400e040 con 0x7f59ec106e80 2024-02-21T23:46:04.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.951+0000 7f59f3922700 1 -- 172.21.15.90:0/1570722742 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59ec1a5ab0 con 0x7f59ec106e80 2024-02-21T23:46:04.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.951+0000 7f59e27fc700 1 -- 172.21.15.90:0/1570722742 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f59e40078d0 con 0x7f59ec106e80 2024-02-21T23:46:04.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.951+0000 7f59e27fc700 1 -- 172.21.15.90:0/1570722742 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e401b450 con 0x7f59ec106e80 2024-02-21T23:46:04.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.951+0000 7f59f3922700 1 -- 172.21.15.90:0/1570722742 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59ec1a5f20 con 0x7f59ec106e80 2024-02-21T23:46:04.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.952+0000 7f59f3922700 1 -- 172.21.15.90:0/1570722742 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59ec109210 con 0x7f59ec106e80 2024-02-21T23:46:04.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.953+0000 7f59e27fc700 1 -- 172.21.15.90:0/1570722742 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7f59e401d580 con 0x7f59ec106e80 2024-02-21T23:46:04.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.953+0000 7f59e27fc700 1 --2- 172.21.15.90:0/1570722742 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f59d80a1120 0x7f59d80a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:04.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.953+0000 7f59f16be700 1 --2- 172.21.15.90:0/1570722742 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f59d80a1120 0x7f59d80a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:04.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.953+0000 7f59e27fc700 1 -- 172.21.15.90:0/1570722742 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f59e40cd800 con 0x7f59ec106e80 2024-02-21T23:46:04.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.954+0000 7f59f16be700 1 --2- 172.21.15.90:0/1570722742 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f59d80a1120 0x7f59d80a35e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f59e8005950 tx=0x7f59e800a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:04.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:04.958+0000 7f59e27fc700 1 -- 172.21.15.90:0/1570722742 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f59e40974c0 con 0x7f59ec106e80 2024-02-21T23:46:05.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:05.151+0000 7f59f3922700 1 -- 172.21.15.90:0/1570722742 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f59ec101220 con 0x7f59ec106e80 2024-02-21T23:46:05.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:05.152+0000 7f59e27fc700 1 -- 172.21.15.90:0/1570722742 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7f59e409aae0 con 0x7f59ec106e80 2024-02-21T23:46:05.136 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-02-21T23:46:05.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:05.154+0000 7f59f3922700 1 -- 172.21.15.90:0/1570722742 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f59d80a1120 msgr2=0x7f59d80a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:05.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:05.154+0000 7f59f3922700 1 --2- 172.21.15.90:0/1570722742 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f59d80a1120 0x7f59d80a35e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f59e8005950 tx=0x7f59e800a300 comp rx=0 tx=0).stop 2024-02-21T23:46:05.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:05.154+0000 7f59f3922700 1 -- 172.21.15.90:0/1570722742 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59ec106e80 msgr2=0x7f59ec100720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:05.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:05.154+0000 7f59f3922700 1 --2- 172.21.15.90:0/1570722742 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59ec106e80 0x7f59ec100720 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f59e400b070 tx=0x7f59e40094c0 comp rx=0 tx=0).stop 2024-02-21T23:46:05.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:05.154+0000 7f59f3922700 1 -- 172.21.15.90:0/1570722742 shutdown_connections 2024-02-21T23:46:05.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:05.154+0000 7f59f3922700 1 --2- 172.21.15.90:0/1570722742 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f59d80a1120 0x7f59d80a35e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:05.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:05.154+0000 7f59f3922700 1 --2- 172.21.15.90:0/1570722742 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59ec106e80 0x7f59ec100720 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:05.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:05.154+0000 7f59f3922700 1 --2- 172.21.15.90:0/1570722742 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59ec068740 0x7f59ec0ffca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:05.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:05.154+0000 7f59f3922700 1 --2- 172.21.15.90:0/1570722742 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59ec069100 0x7f59ec1001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:05.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:05.154+0000 7f59f3922700 1 -- 172.21.15.90:0/1570722742 >> 172.21.15.90:0/1570722742 conn(0x7f59ec076d70 msgr2=0x7f59ec0fe8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:05.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:05.154+0000 7f59f3922700 1 -- 172.21.15.90:0/1570722742 shutdown_connections 2024-02-21T23:46:05.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:05.154+0000 7f59f3922700 1 -- 172.21.15.90:0/1570722742 wait complete. 2024-02-21T23:46:05.457 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:05 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/1570722742' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-21T23:46:05.507 INFO:tasks.cephadm:Adding mds.a on smithi090 2024-02-21T23:46:05.507 INFO:tasks.cephadm:Adding mds.d on smithi090 2024-02-21T23:46:05.507 INFO:tasks.cephadm:Adding mds.g on smithi090 2024-02-21T23:46:05.507 INFO:tasks.cephadm:Adding mds.j on smithi090 2024-02-21T23:46:05.507 INFO:tasks.cephadm:Adding mds.b on smithi176 2024-02-21T23:46:05.507 INFO:tasks.cephadm:Adding mds.e on smithi176 2024-02-21T23:46:05.508 INFO:tasks.cephadm:Adding mds.h on smithi176 2024-02-21T23:46:05.508 INFO:tasks.cephadm:Adding mds.k on smithi176 2024-02-21T23:46:05.508 INFO:tasks.cephadm:Adding mds.c on smithi178 2024-02-21T23:46:05.508 INFO:tasks.cephadm:Adding mds.f on smithi178 2024-02-21T23:46:05.508 INFO:tasks.cephadm:Adding mds.i on smithi178 2024-02-21T23:46:05.508 INFO:tasks.cephadm:Adding mds.l on smithi178 2024-02-21T23:46:05.508 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph orch apply mds all '12;smithi090=a;smithi090=d;smithi090=g;smithi090=j;smithi176=b;smithi176=e;smithi176=h;smithi176=k;smithi178=c;smithi178=f;smithi178=i;smithi178=l' 2024-02-21T23:46:05.542 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:05 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/1570722742' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-21T23:46:05.583 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:05 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/1570722742' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-21T23:46:05.743 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.c/config 2024-02-21T23:46:06.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:06 smithi090 ceph-mon[43385]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:06.583 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:06 smithi176 ceph-mon[47172]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:06.640 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:06 smithi178 ceph-mon[47547]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:06.836 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.850+0000 7f1e51163700 1 -- 172.21.15.178:0/13865114 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1e4c100e90 msgr2=0x7f1e4c1012f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:06.837 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.850+0000 7f1e51163700 1 --2- 172.21.15.178:0/13865114 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1e4c100e90 0x7f1e4c1012f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c009990 tx=0x7f1e3c009ca0 comp rx=0 tx=0).stop 2024-02-21T23:46:06.837 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.850+0000 7f1e51163700 1 -- 172.21.15.178:0/13865114 shutdown_connections 2024-02-21T23:46:06.837 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.850+0000 7f1e51163700 1 --2- 172.21.15.178:0/13865114 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e4c101830 0x7f1e4c1080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:06.837 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.850+0000 7f1e51163700 1 --2- 172.21.15.178:0/13865114 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e4c0ffc90 0x7f1e4c1000b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:06.837 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.850+0000 7f1e51163700 1 --2- 172.21.15.178:0/13865114 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1e4c100e90 0x7f1e4c1012f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:06.837 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.850+0000 7f1e51163700 1 -- 172.21.15.178:0/13865114 >> 172.21.15.178:0/13865114 conn(0x7f1e4c0fb230 msgr2=0x7f1e4c0fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:06.837 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.850+0000 7f1e51163700 1 -- 172.21.15.178:0/13865114 shutdown_connections 2024-02-21T23:46:06.837 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.850+0000 7f1e51163700 1 -- 172.21.15.178:0/13865114 wait complete. 2024-02-21T23:46:06.837 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.850+0000 7f1e51163700 1 Processor -- start 2024-02-21T23:46:06.837 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.851+0000 7f1e51163700 1 -- start start 2024-02-21T23:46:06.837 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.851+0000 7f1e51163700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e4c0ffc90 0x7f1e4c195ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:06.837 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.851+0000 7f1e51163700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1e4c100e90 0x7f1e4c196010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:06.838 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.851+0000 7f1e51163700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e4c101830 0x7f1e4c19d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:06.838 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.851+0000 7f1e51163700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4c19d610 con 0x7f1e4c0ffc90 2024-02-21T23:46:06.838 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.851+0000 7f1e51163700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4c19d750 con 0x7f1e4c101830 2024-02-21T23:46:06.838 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.851+0000 7f1e51163700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4c19d8e0 con 0x7f1e4c100e90 2024-02-21T23:46:06.838 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.851+0000 7f1e4a59c700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1e4c100e90 0x7f1e4c196010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:06.838 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.851+0000 7f1e4a59c700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1e4c100e90 0x7f1e4c196010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:40258/0 (socket says 172.21.15.178:40258) 2024-02-21T23:46:06.838 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.851+0000 7f1e4a59c700 1 -- 172.21.15.178:0/902008038 learned_addr learned my addr 172.21.15.178:0/902008038 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-21T23:46:06.838 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.851+0000 7f1e4ad9d700 1 --2- 172.21.15.178:0/902008038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e4c0ffc90 0x7f1e4c195ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:06.838 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.851+0000 7f1e4b59e700 1 --2- 172.21.15.178:0/902008038 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e4c101830 0x7f1e4c19d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:06.838 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.851+0000 7f1e4a59c700 1 -- 172.21.15.178:0/902008038 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e4c101830 msgr2=0x7f1e4c19d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:06.838 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.851+0000 7f1e4a59c700 1 --2- 172.21.15.178:0/902008038 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e4c101830 0x7f1e4c19d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:06.838 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.851+0000 7f1e4a59c700 1 -- 172.21.15.178:0/902008038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e4c0ffc90 msgr2=0x7f1e4c195ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:06.839 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.851+0000 7f1e4a59c700 1 --2- 172.21.15.178:0/902008038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e4c0ffc90 0x7f1e4c195ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:06.839 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.851+0000 7f1e4a59c700 1 -- 172.21.15.178:0/902008038 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e3c0095f0 con 0x7f1e4c100e90 2024-02-21T23:46:06.839 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.851+0000 7f1e4b59e700 1 --2- 172.21.15.178:0/902008038 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e4c101830 0x7f1e4c19d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:46:06.839 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.851+0000 7f1e4ad9d700 1 --2- 172.21.15.178:0/902008038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e4c0ffc90 0x7f1e4c195ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:46:06.839 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.851+0000 7f1e4a59c700 1 --2- 172.21.15.178:0/902008038 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1e4c100e90 0x7f1e4c196010 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c00b5c0 tx=0x7f1e3c009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:06.839 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.851+0000 7f1e33fff700 1 -- 172.21.15.178:0/902008038 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c00fd90 con 0x7f1e4c100e90 2024-02-21T23:46:06.839 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.852+0000 7f1e51163700 1 -- 172.21.15.178:0/902008038 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e4c0686e0 con 0x7f1e4c100e90 2024-02-21T23:46:06.839 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.852+0000 7f1e33fff700 1 -- 172.21.15.178:0/902008038 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1e3c003e20 con 0x7f1e4c100e90 2024-02-21T23:46:06.839 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.852+0000 7f1e33fff700 1 -- 172.21.15.178:0/902008038 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c0206a0 con 0x7f1e4c100e90 2024-02-21T23:46:06.839 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.852+0000 7f1e51163700 1 -- 172.21.15.178:0/902008038 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e4c068bd0 con 0x7f1e4c100e90 2024-02-21T23:46:06.840 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.853+0000 7f1e51163700 1 -- 172.21.15.178:0/902008038 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e4c04ea40 con 0x7f1e4c100e90 2024-02-21T23:46:06.840 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.853+0000 7f1e33fff700 1 -- 172.21.15.178:0/902008038 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7f1e3c005160 con 0x7f1e4c100e90 2024-02-21T23:46:06.840 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.854+0000 7f1e33fff700 1 --2- 172.21.15.178:0/902008038 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f1e340a11a0 0x7f1e340a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:06.840 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.854+0000 7f1e33fff700 1 -- 172.21.15.178:0/902008038 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f1e3c0cf840 con 0x7f1e4c100e90 2024-02-21T23:46:06.840 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.854+0000 7f1e4ad9d700 1 --2- 172.21.15.178:0/902008038 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f1e340a11a0 0x7f1e340a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:06.840 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.855+0000 7f1e4ad9d700 1 --2- 172.21.15.178:0/902008038 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f1e340a11a0 0x7f1e340a3660 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1e40005950 tx=0x7f1e400058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:06.843 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:06.858+0000 7f1e33fff700 1 -- 172.21.15.178:0/902008038 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f1e3c099460 con 0x7f1e4c100e90 2024-02-21T23:46:07.044 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:07.059+0000 7f1e51163700 1 -- 172.21.15.178:0/902008038 --> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi090=a;smithi090=d;smithi090=g;smithi090=j;smithi176=b;smithi176=e;smithi176=h;smithi176=k;smithi178=c;smithi178=f;smithi178=i;smithi178=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f1e4c068df0 con 0x7f1e340a11a0 2024-02-21T23:46:07.054 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:07.070+0000 7f1e33fff700 1 -- 172.21.15.178:0/902008038 <== mgr.14150 v2:172.21.15.90:6800/3697394685 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f1e4c068df0 con 0x7f1e340a11a0 2024-02-21T23:46:07.054 INFO:teuthology.orchestra.run.smithi178.stdout:Scheduled mds.all update... 2024-02-21T23:46:07.057 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:07.072+0000 7f1e51163700 1 -- 172.21.15.178:0/902008038 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f1e340a11a0 msgr2=0x7f1e340a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:07.057 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:07.072+0000 7f1e51163700 1 --2- 172.21.15.178:0/902008038 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f1e340a11a0 0x7f1e340a3660 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1e40005950 tx=0x7f1e400058e0 comp rx=0 tx=0).stop 2024-02-21T23:46:07.057 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:07.072+0000 7f1e51163700 1 -- 172.21.15.178:0/902008038 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1e4c100e90 msgr2=0x7f1e4c196010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:07.057 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:07.072+0000 7f1e51163700 1 --2- 172.21.15.178:0/902008038 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1e4c100e90 0x7f1e4c196010 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c00b5c0 tx=0x7f1e3c009f90 comp rx=0 tx=0).stop 2024-02-21T23:46:07.057 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:07.072+0000 7f1e51163700 1 -- 172.21.15.178:0/902008038 shutdown_connections 2024-02-21T23:46:07.057 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:07.072+0000 7f1e51163700 1 --2- 172.21.15.178:0/902008038 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f1e340a11a0 0x7f1e340a3660 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:07.057 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:07.072+0000 7f1e51163700 1 --2- 172.21.15.178:0/902008038 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e4c101830 0x7f1e4c19d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:07.057 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:07.072+0000 7f1e51163700 1 --2- 172.21.15.178:0/902008038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e4c0ffc90 0x7f1e4c195ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:07.057 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:07.072+0000 7f1e51163700 1 --2- 172.21.15.178:0/902008038 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1e4c100e90 0x7f1e4c196010 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:07.057 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:07.072+0000 7f1e51163700 1 -- 172.21.15.178:0/902008038 >> 172.21.15.178:0/902008038 conn(0x7f1e4c0fb230 msgr2=0x7f1e4c1041e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:07.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:07.072+0000 7f1e51163700 1 -- 172.21.15.178:0/902008038 shutdown_connections 2024-02-21T23:46:07.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-21T23:46:07.072+0000 7f1e51163700 1 -- 172.21.15.178:0/902008038 wait complete. 2024-02-21T23:46:07.501 DEBUG:teuthology.orchestra.run.smithi090:mds.a> sudo journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@mds.a.service 2024-02-21T23:46:07.503 DEBUG:teuthology.orchestra.run.smithi090:mds.d> sudo journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@mds.d.service 2024-02-21T23:46:07.533 INFO:journalctl@ceph.mds.a.smithi090.stdout:-- Logs begin at Wed 2024-02-21 23:24:49 UTC. -- 2024-02-21T23:46:07.534 DEBUG:teuthology.orchestra.run.smithi090:mds.g> sudo journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@mds.g.service 2024-02-21T23:46:07.560 INFO:journalctl@ceph.mds.d.smithi090.stdout:-- Logs begin at Wed 2024-02-21 23:24:49 UTC. -- 2024-02-21T23:46:07.563 DEBUG:teuthology.orchestra.run.smithi090:mds.j> sudo journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@mds.j.service 2024-02-21T23:46:07.587 INFO:journalctl@ceph.mds.g.smithi090.stdout:-- Logs begin at Wed 2024-02-21 23:24:49 UTC. -- 2024-02-21T23:46:07.588 DEBUG:teuthology.orchestra.run.smithi176:mds.b> sudo journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@mds.b.service 2024-02-21T23:46:07.590 DEBUG:teuthology.orchestra.run.smithi176:mds.e> sudo journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@mds.e.service 2024-02-21T23:46:07.613 INFO:journalctl@ceph.mds.j.smithi090.stdout:-- Logs begin at Wed 2024-02-21 23:24:49 UTC. -- 2024-02-21T23:46:07.629 INFO:journalctl@ceph.mds.b.smithi176.stdout:-- Logs begin at Wed 2024-02-21 23:23:58 UTC. -- 2024-02-21T23:46:07.630 DEBUG:teuthology.orchestra.run.smithi176:mds.h> sudo journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@mds.h.service 2024-02-21T23:46:07.660 INFO:journalctl@ceph.mds.e.smithi176.stdout:-- Logs begin at Wed 2024-02-21 23:23:58 UTC. -- 2024-02-21T23:46:07.661 DEBUG:teuthology.orchestra.run.smithi176:mds.k> sudo journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@mds.k.service 2024-02-21T23:46:07.689 INFO:journalctl@ceph.mds.h.smithi176.stdout:-- Logs begin at Wed 2024-02-21 23:23:58 UTC. -- 2024-02-21T23:46:07.690 DEBUG:teuthology.orchestra.run.smithi178:mds.c> sudo journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@mds.c.service 2024-02-21T23:46:07.692 DEBUG:teuthology.orchestra.run.smithi178:mds.f> sudo journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@mds.f.service 2024-02-21T23:46:07.717 INFO:journalctl@ceph.mds.k.smithi176.stdout:-- Logs begin at Wed 2024-02-21 23:23:58 UTC. -- 2024-02-21T23:46:07.723 INFO:journalctl@ceph.mds.c.smithi178.stdout:-- Logs begin at Wed 2024-02-21 23:24:26 UTC. -- 2024-02-21T23:46:07.724 DEBUG:teuthology.orchestra.run.smithi178:mds.i> sudo journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@mds.i.service 2024-02-21T23:46:07.751 INFO:journalctl@ceph.mds.f.smithi178.stdout:-- Logs begin at Wed 2024-02-21 23:24:26 UTC. -- 2024-02-21T23:46:07.752 DEBUG:teuthology.orchestra.run.smithi178:mds.l> sudo journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@mds.l.service 2024-02-21T23:46:07.778 INFO:journalctl@ceph.mds.i.smithi178.stdout:-- Logs begin at Wed 2024-02-21 23:24:26 UTC. -- 2024-02-21T23:46:07.779 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-02-21T23:46:07.780 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:46:07.809 INFO:journalctl@ceph.mds.l.smithi178.stdout:-- Logs begin at Wed 2024-02-21 23:24:26 UTC. -- 2024-02-21T23:46:07.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.935+0000 7f309c788700 1 -- 172.21.15.90:0/2849711143 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30941068f0 msgr2=0x7f3094108d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:07.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.935+0000 7f309c788700 1 --2- 172.21.15.90:0/2849711143 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30941068f0 0x7f3094108d60 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f308c00b0a0 tx=0x7f308c00b3b0 comp rx=0 tx=0).stop 2024-02-21T23:46:07.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.935+0000 7f309c788700 1 -- 172.21.15.90:0/2849711143 shutdown_connections 2024-02-21T23:46:07.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.935+0000 7f309c788700 1 --2- 172.21.15.90:0/2849711143 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30941068f0 0x7f3094108d60 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:07.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.935+0000 7f309c788700 1 --2- 172.21.15.90:0/2849711143 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3094101690 0x7f3094103a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:07.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.935+0000 7f309c788700 1 --2- 172.21.15.90:0/2849711143 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3094103fc0 0x7f30941063b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:07.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.935+0000 7f309c788700 1 -- 172.21.15.90:0/2849711143 >> 172.21.15.90:0/2849711143 conn(0x7f30940fb080 msgr2=0x7f30940fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:07.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.935+0000 7f309c788700 1 -- 172.21.15.90:0/2849711143 shutdown_connections 2024-02-21T23:46:07.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.935+0000 7f309c788700 1 -- 172.21.15.90:0/2849711143 wait complete. 2024-02-21T23:46:07.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.936+0000 7f309c788700 1 Processor -- start 2024-02-21T23:46:07.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.936+0000 7f309c788700 1 -- start start 2024-02-21T23:46:07.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.936+0000 7f309c788700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3094101690 0x7f3094196c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:07.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.936+0000 7f309c788700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3094103fc0 0x7f3094197160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:07.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.936+0000 7f309c788700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f30941068f0 0x7f309419e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:07.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.936+0000 7f309c788700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309419e750 con 0x7f3094103fc0 2024-02-21T23:46:07.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.936+0000 7f309c788700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309419e890 con 0x7f30941068f0 2024-02-21T23:46:07.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.936+0000 7f309c788700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309419ea20 con 0x7f3094101690 2024-02-21T23:46:07.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.936+0000 7f3099d23700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3094103fc0 0x7f3094197160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:07.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.936+0000 7f3099d23700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3094103fc0 0x7f3094197160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45720/0 (socket says 172.21.15.90:45720) 2024-02-21T23:46:07.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.936+0000 7f3099d23700 1 -- 172.21.15.90:0/2334908623 learned_addr learned my addr 172.21.15.90:0/2334908623 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:07.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.936+0000 7f309a524700 1 --2- 172.21.15.90:0/2334908623 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3094101690 0x7f3094196c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:07.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.936+0000 7f309ad25700 1 --2- 172.21.15.90:0/2334908623 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f30941068f0 0x7f309419e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:07.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.936+0000 7f3099d23700 1 -- 172.21.15.90:0/2334908623 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3094101690 msgr2=0x7f3094196c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:07.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.936+0000 7f3099d23700 1 --2- 172.21.15.90:0/2334908623 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3094101690 0x7f3094196c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:07.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.936+0000 7f3099d23700 1 -- 172.21.15.90:0/2334908623 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f30941068f0 msgr2=0x7f309419e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:07.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.936+0000 7f3099d23700 1 --2- 172.21.15.90:0/2334908623 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f30941068f0 0x7f309419e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:07.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.936+0000 7f3099d23700 1 -- 172.21.15.90:0/2334908623 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f308c009d00 con 0x7f3094103fc0 2024-02-21T23:46:07.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.936+0000 7f3099d23700 1 --2- 172.21.15.90:0/2334908623 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3094103fc0 0x7f3094197160 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f309000d950 tx=0x7f309000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:07.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.937+0000 7f308b7fe700 1 -- 172.21.15.90:0/2334908623 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309000f890 con 0x7f3094103fc0 2024-02-21T23:46:07.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.937+0000 7f308b7fe700 1 -- 172.21.15.90:0/2334908623 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3090010460 con 0x7f3094103fc0 2024-02-21T23:46:07.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.937+0000 7f309c788700 1 -- 172.21.15.90:0/2334908623 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f309419ed00 con 0x7f3094103fc0 2024-02-21T23:46:07.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.937+0000 7f308b7fe700 1 -- 172.21.15.90:0/2334908623 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309000e610 con 0x7f3094103fc0 2024-02-21T23:46:07.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.937+0000 7f309c788700 1 -- 172.21.15.90:0/2334908623 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f309419f200 con 0x7f3094103fc0 2024-02-21T23:46:07.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.937+0000 7f309c788700 1 -- 172.21.15.90:0/2334908623 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30940fcc00 con 0x7f3094103fc0 2024-02-21T23:46:07.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.939+0000 7f308b7fe700 1 -- 172.21.15.90:0/2334908623 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7f30900108d0 con 0x7f3094103fc0 2024-02-21T23:46:07.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.939+0000 7f308b7fe700 1 --2- 172.21.15.90:0/2334908623 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f30800a11a0 0x7f30800a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:07.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.939+0000 7f309a524700 1 --2- 172.21.15.90:0/2334908623 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f30800a11a0 0x7f30800a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:07.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.939+0000 7f308b7fe700 1 -- 172.21.15.90:0/2334908623 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f30900cc150 con 0x7f3094103fc0 2024-02-21T23:46:07.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.940+0000 7f309a524700 1 --2- 172.21.15.90:0/2334908623 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f30800a11a0 0x7f30800a3660 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3084005950 tx=0x7f308400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:07.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:07.943+0000 7f308b7fe700 1 -- 172.21.15.90:0/2334908623 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f3090095d80 con 0x7f3094103fc0 2024-02-21T23:46:08.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.145+0000 7f309c788700 1 -- 172.21.15.90:0/2334908623 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3094067110 con 0x7f3094103fc0 2024-02-21T23:46:08.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.146+0000 7f308b7fe700 1 -- 172.21.15.90:0/2334908623 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+18387 (secure 0 0 0) 0x7f30900993a0 con 0x7f3094103fc0 2024-02-21T23:46:08.130 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:46:08.130 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":77,"fsid":"7ed81218-d112-11ee-95bd-87774f69a715","created":"2024-02-21T23:40:41.785526+0000","modified":"2024-02-21T23:46:01.867321+0000","last_up_change":"2024-02-21T23:46:00.860501+0000","last_in_change":"2024-02-21T23:45:44.423815+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:43:21.675789+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"77878890-d9a5-4d5b-a943-fec29ee0446e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6803","nonce":885319826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6805","nonce":885319826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6809","nonce":885319826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6807","nonce":885319826}]},"public_addr":"172.21.15.90:6803/885319826","cluster_addr":"172.21.15.90:6805/885319826","heartbeat_back_addr":"172.21.15.90:6809/885319826","heartbeat_front_addr":"172.21.15.90:6807/885319826","state":["exists","up"]},{"osd":1,"uuid":"7d6ea950-7a57-4f13-ab38-10a658912783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6801","nonce":4085240925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6803","nonce":4085240925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6807","nonce":4085240925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6805","nonce":4085240925}]},"public_addr":"172.21.15.176:6801/4085240925","cluster_addr":"172.21.15.176:6803/4085240925","heartbeat_back_addr":"172.21.15.176:6807/4085240925","heartbeat_front_addr":"172.21.15.176:6805/4085240925","state":["exists","up"]},{"osd":2,"uuid":"b83486bb-27f2-4d56-97fb-fb336fec4b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6801","nonce":3597250789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6803","nonce":3597250789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6807","nonce":3597250789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6805","nonce":3597250789}]},"public_addr":"172.21.15.178:6801/3597250789","cluster_addr":"172.21.15.178:6803/3597250789","heartbeat_back_addr":"172.21.15.178:6807/3597250789","heartbeat_front_addr":"172.21.15.178:6805/3597250789","state":["exists","up"]},{"osd":3,"uuid":"870e5bb6-eed5-4a0a-8e03-38de4f816b44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6811","nonce":3299043821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6813","nonce":3299043821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6817","nonce":3299043821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6815","nonce":3299043821}]},"public_addr":"172.21.15.90:6811/3299043821","cluster_addr":"172.21.15.90:6813/3299043821","heartbeat_back_addr":"172.21.15.90:6817/3299043821","heartbeat_front_addr":"172.21.15.90:6815/3299043821","state":["exists","up"]},{"osd":4,"uuid":"cb9f7f48-76e0-4368-ab87-2a09a8487ccd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6809","nonce":3548915687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6811","nonce":3548915687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6815","nonce":3548915687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6813","nonce":3548915687}]},"public_addr":"172.21.15.176:6809/3548915687","cluster_addr":"172.21.15.176:6811/3548915687","heartbeat_back_addr":"172.21.15.176:6815/3548915687","heartbeat_front_addr":"172.21.15.176:6813/3548915687","state":["exists","up"]},{"osd":5,"uuid":"54749022-c50d-4831-a1d6-8412b59c8a01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6809","nonce":286274518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6811","nonce":286274518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6815","nonce":286274518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6813","nonce":286274518}]},"public_addr":"172.21.15.178:6809/286274518","cluster_addr":"172.21.15.178:6811/286274518","heartbeat_back_addr":"172.21.15.178:6815/286274518","heartbeat_front_addr":"172.21.15.178:6813/286274518","state":["exists","up"]},{"osd":6,"uuid":"09bb5393-2efd-4a63-b0e8-99bef5bced13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6819","nonce":67640620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6821","nonce":67640620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6825","nonce":67640620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6823","nonce":67640620}]},"public_addr":"172.21.15.90:6819/67640620","cluster_addr":"172.21.15.90:6821/67640620","heartbeat_back_addr":"172.21.15.90:6825/67640620","heartbeat_front_addr":"172.21.15.90:6823/67640620","state":["exists","up"]},{"osd":7,"uuid":"7a859e69-6b63-467b-a3b3-99247d66f5c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6817","nonce":3998561598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6819","nonce":3998561598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6823","nonce":3998561598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6821","nonce":3998561598}]},"public_addr":"172.21.15.176:6817/3998561598","cluster_addr":"172.21.15.176:6819/3998561598","heartbeat_back_addr":"172.21.15.176:6823/3998561598","heartbeat_front_addr":"172.21.15.176:6821/3998561598","state":["exists","up"]},{"osd":8,"uuid":"bc23144e-22a5-4894-a8b5-9323383574d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6817","nonce":490229963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6819","nonce":490229963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6823","nonce":490229963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6821","nonce":490229963}]},"public_addr":"172.21.15.178:6817/490229963","cluster_addr":"172.21.15.178:6819/490229963","heartbeat_back_addr":"172.21.15.178:6823/490229963","heartbeat_front_addr":"172.21.15.178:6821/490229963","state":["exists","up"]},{"osd":9,"uuid":"0c2cecd4-55de-4fc4-a6f4-ca97c8460444","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6827","nonce":2061394761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6829","nonce":2061394761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6833","nonce":2061394761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6831","nonce":2061394761}]},"public_addr":"172.21.15.90:6827/2061394761","cluster_addr":"172.21.15.90:6829/2061394761","heartbeat_back_addr":"172.21.15.90:6833/2061394761","heartbeat_front_addr":"172.21.15.90:6831/2061394761","state":["exists","up"]},{"osd":10,"uuid":"fa04232b-b1ec-447a-8e94-7c9bf26d0291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6825","nonce":639411757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6827","nonce":639411757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6831","nonce":639411757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6829","nonce":639411757}]},"public_addr":"172.21.15.176:6825/639411757","cluster_addr":"172.21.15.176:6827/639411757","heartbeat_back_addr":"172.21.15.176:6831/639411757","heartbeat_front_addr":"172.21.15.176:6829/639411757","state":["exists","up"]},{"osd":11,"uuid":"7282e294-64fc-4856-9c74-86115f69408c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6825","nonce":4144536256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6827","nonce":4144536256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6831","nonce":4144536256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6829","nonce":4144536256}]},"public_addr":"172.21.15.178:6825/4144536256","cluster_addr":"172.21.15.178:6827/4144536256","heartbeat_back_addr":"172.21.15.178:6831/4144536256","heartbeat_front_addr":"172.21.15.178:6829/4144536256","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:42:44.280345+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:01.235630+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:19.408896+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:36.201749+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:53.401854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:44:11.342158+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:44:29.324502+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:44:46.821595+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:04.642344+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:23.019004+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:41.140149+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:59.401951+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/159655312":"2024-02-22T23:41:33.360021+0000","172.21.15.90:6801/1385551604":"2024-02-22T23:41:33.360021+0000","172.21.15.90:0/1452574770":"2024-02-22T23:41:33.360021+0000","172.21.15.90:0/3815292343":"2024-02-22T23:41:11.484145+0000","172.21.15.90:6800/1385551604":"2024-02-22T23:41:33.360021+0000","172.21.15.90:6801/4247695574":"2024-02-22T23:41:11.484145+0000","172.21.15.90:0/1328266622":"2024-02-22T23:41:11.484145+0000","172.21.15.90:6800/4247695574":"2024-02-22T23:41:11.484145+0000","172.21.15.90:0/2569421522":"2024-02-22T23:41:11.484145+0000","172.21.15.90:0/3019701027":"2024-02-22T23:41:33.360021+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:46:08.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.148+0000 7f309c788700 1 -- 172.21.15.90:0/2334908623 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f30800a11a0 msgr2=0x7f30800a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:08.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.148+0000 7f309c788700 1 --2- 172.21.15.90:0/2334908623 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f30800a11a0 0x7f30800a3660 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3084005950 tx=0x7f308400b410 comp rx=0 tx=0).stop 2024-02-21T23:46:08.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.148+0000 7f309c788700 1 -- 172.21.15.90:0/2334908623 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3094103fc0 msgr2=0x7f3094197160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:08.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.148+0000 7f309c788700 1 --2- 172.21.15.90:0/2334908623 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3094103fc0 0x7f3094197160 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f309000d950 tx=0x7f309000dc60 comp rx=0 tx=0).stop 2024-02-21T23:46:08.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.148+0000 7f309c788700 1 -- 172.21.15.90:0/2334908623 shutdown_connections 2024-02-21T23:46:08.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.148+0000 7f309c788700 1 --2- 172.21.15.90:0/2334908623 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f30800a11a0 0x7f30800a3660 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:08.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.148+0000 7f309c788700 1 --2- 172.21.15.90:0/2334908623 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f30941068f0 0x7f309419e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:08.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.148+0000 7f309c788700 1 --2- 172.21.15.90:0/2334908623 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3094101690 0x7f3094196c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:08.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.148+0000 7f309c788700 1 --2- 172.21.15.90:0/2334908623 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3094103fc0 0x7f3094197160 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:08.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.148+0000 7f309c788700 1 -- 172.21.15.90:0/2334908623 >> 172.21.15.90:0/2334908623 conn(0x7f30940fb080 msgr2=0x7f3094101fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:08.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.149+0000 7f309c788700 1 -- 172.21.15.90:0/2334908623 shutdown_connections 2024-02-21T23:46:08.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.149+0000 7f309c788700 1 -- 172.21.15.90:0/2334908623 wait complete. 2024-02-21T23:46:08.145 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:43:21.675789+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:46:08.145 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:46:08.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.286+0000 7f7445303700 1 -- 172.21.15.90:0/3981762110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74401017a0 msgr2=0x7f7440108050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:08.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.286+0000 7f7445303700 1 --2- 172.21.15.90:0/3981762110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74401017a0 0x7f7440108050 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f743800b0d0 tx=0x7f743800b3e0 comp rx=0 tx=0).stop 2024-02-21T23:46:08.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.286+0000 7f7445303700 1 -- 172.21.15.90:0/3981762110 shutdown_connections 2024-02-21T23:46:08.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.286+0000 7f7445303700 1 --2- 172.21.15.90:0/3981762110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74401017a0 0x7f7440108050 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:08.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.286+0000 7f7445303700 1 --2- 172.21.15.90:0/3981762110 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f74400ffc10 0x7f7440100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:08.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.286+0000 7f7445303700 1 --2- 172.21.15.90:0/3981762110 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7440100e00 0x7f7440101260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:08.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.286+0000 7f7445303700 1 -- 172.21.15.90:0/3981762110 >> 172.21.15.90:0/3981762110 conn(0x7f74400fb180 msgr2=0x7f74400fd5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:08.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.286+0000 7f7445303700 1 -- 172.21.15.90:0/3981762110 shutdown_connections 2024-02-21T23:46:08.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.286+0000 7f7445303700 1 -- 172.21.15.90:0/3981762110 wait complete. 2024-02-21T23:46:08.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.287+0000 7f7445303700 1 Processor -- start 2024-02-21T23:46:08.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.287+0000 7f7445303700 1 -- start start 2024-02-21T23:46:08.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.287+0000 7f7445303700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f74400ffc10 0x7f7440196d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:08.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.287+0000 7f7445303700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7440100e00 0x7f74401972c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:08.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.287+0000 7f7445303700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f74401017a0 0x7f744019e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:08.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.287+0000 7f7445303700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f744019e870 con 0x7f7440100e00 2024-02-21T23:46:08.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.287+0000 7f7445303700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f744019e9e0 con 0x7f74400ffc10 2024-02-21T23:46:08.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.287+0000 7f7445303700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f744019eba0 con 0x7f74401017a0 2024-02-21T23:46:08.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.287+0000 7f743e7fc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7440100e00 0x7f74401972c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:08.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.287+0000 7f743e7fc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7440100e00 0x7f74401972c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49426/0 (socket says 172.21.15.90:49426) 2024-02-21T23:46:08.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.287+0000 7f743e7fc700 1 -- 172.21.15.90:0/4216071913 learned_addr learned my addr 172.21.15.90:0/4216071913 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:08.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.287+0000 7f743f7fe700 1 --2- 172.21.15.90:0/4216071913 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f74401017a0 0x7f744019e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:08.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.287+0000 7f743effd700 1 --2- 172.21.15.90:0/4216071913 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f74400ffc10 0x7f7440196d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:08.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.288+0000 7f743e7fc700 1 -- 172.21.15.90:0/4216071913 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f74401017a0 msgr2=0x7f744019e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:08.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.288+0000 7f743e7fc700 1 --2- 172.21.15.90:0/4216071913 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f74401017a0 0x7f744019e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:08.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.288+0000 7f743e7fc700 1 -- 172.21.15.90:0/4216071913 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f74400ffc10 msgr2=0x7f7440196d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:08.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.288+0000 7f743e7fc700 1 --2- 172.21.15.90:0/4216071913 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f74400ffc10 0x7f7440196d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:08.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.288+0000 7f743e7fc700 1 -- 172.21.15.90:0/4216071913 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7438009d00 con 0x7f7440100e00 2024-02-21T23:46:08.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.288+0000 7f743effd700 1 --2- 172.21.15.90:0/4216071913 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f74400ffc10 0x7f7440196d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:46:08.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.288+0000 7f743e7fc700 1 --2- 172.21.15.90:0/4216071913 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7440100e00 0x7f74401972c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f743400d950 tx=0x7f743400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:08.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.288+0000 7f7427fff700 1 -- 172.21.15.90:0/4216071913 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7434009940 con 0x7f7440100e00 2024-02-21T23:46:08.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.288+0000 7f7445303700 1 -- 172.21.15.90:0/4216071913 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f744019ee80 con 0x7f7440100e00 2024-02-21T23:46:08.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.288+0000 7f7427fff700 1 -- 172.21.15.90:0/4216071913 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7434010460 con 0x7f7440100e00 2024-02-21T23:46:08.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.288+0000 7f7427fff700 1 -- 172.21.15.90:0/4216071913 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743400f620 con 0x7f7440100e00 2024-02-21T23:46:08.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.288+0000 7f7445303700 1 -- 172.21.15.90:0/4216071913 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f744019f300 con 0x7f7440100e00 2024-02-21T23:46:08.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.289+0000 7f7445303700 1 -- 172.21.15.90:0/4216071913 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7440067ed0 con 0x7f7440100e00 2024-02-21T23:46:08.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.290+0000 7f7427fff700 1 -- 172.21.15.90:0/4216071913 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7f74340108d0 con 0x7f7440100e00 2024-02-21T23:46:08.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.290+0000 7f7427fff700 1 --2- 172.21.15.90:0/4216071913 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f74280a1120 0x7f74280a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:08.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.290+0000 7f743effd700 1 --2- 172.21.15.90:0/4216071913 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f74280a1120 0x7f74280a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:08.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.290+0000 7f7427fff700 1 -- 172.21.15.90:0/4216071913 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f74340cc2e0 con 0x7f7440100e00 2024-02-21T23:46:08.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.291+0000 7f743effd700 1 --2- 172.21.15.90:0/4216071913 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f74280a1120 0x7f74280a35e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7430005fd0 tx=0x7f7430005dd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:08.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.294+0000 7f7427fff700 1 -- 172.21.15.90:0/4216071913 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f7434095f10 con 0x7f7440100e00 2024-02-21T23:46:08.328 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:08 smithi178 ceph-mon[47547]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi090=a;smithi090=d;smithi090=g;smithi090=j;smithi176=b;smithi176=e;smithi176=h;smithi176=k;smithi178=c;smithi178=f;smithi178=i;smithi178=l", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:46:08.328 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:08 smithi178 ceph-mon[47547]: Saving service mds.all spec with placement smithi090=a;smithi090=d;smithi090=g;smithi090=j;smithi176=b;smithi176=e;smithi176=h;smithi176=k;smithi178=c;smithi178=f;smithi178=i;smithi178=l;count:12 2024-02-21T23:46:08.328 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:08 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:08.328 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:08 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:46:08.328 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:08 smithi178 ceph-mon[47547]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:08.328 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:08 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:08.328 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:08 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:46:08.328 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:08 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:08.329 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:08 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-21T23:46:08.329 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:08 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-21T23:46:08.329 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:08 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:08.329 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:08 smithi178 ceph-mon[47547]: Deploying daemon mds.l on smithi178 2024-02-21T23:46:08.333 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:08 smithi176 ceph-mon[47172]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi090=a;smithi090=d;smithi090=g;smithi090=j;smithi176=b;smithi176=e;smithi176=h;smithi176=k;smithi178=c;smithi178=f;smithi178=i;smithi178=l", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:46:08.333 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:08 smithi176 ceph-mon[47172]: Saving service mds.all spec with placement smithi090=a;smithi090=d;smithi090=g;smithi090=j;smithi176=b;smithi176=e;smithi176=h;smithi176=k;smithi178=c;smithi178=f;smithi178=i;smithi178=l;count:12 2024-02-21T23:46:08.333 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:08 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:08.333 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:08 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:46:08.333 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:08 smithi176 ceph-mon[47172]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:08.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:08 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:08.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:08 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:46:08.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:08 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:08.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:08 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-21T23:46:08.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:08 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-21T23:46:08.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:08 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:08.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:08 smithi176 ceph-mon[47172]: Deploying daemon mds.l on smithi178 2024-02-21T23:46:08.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.487+0000 7f7445303700 1 -- 172.21.15.90:0/4216071913 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f744019f5e0 con 0x7f7440100e00 2024-02-21T23:46:08.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.488+0000 7f7427fff700 1 -- 172.21.15.90:0/4216071913 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7434099530 con 0x7f7440100e00 2024-02-21T23:46:08.471 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-02-21T23:46:08.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.490+0000 7f7445303700 1 -- 172.21.15.90:0/4216071913 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f74280a1120 msgr2=0x7f74280a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:08.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.490+0000 7f7445303700 1 --2- 172.21.15.90:0/4216071913 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f74280a1120 0x7f74280a35e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7430005fd0 tx=0x7f7430005dd0 comp rx=0 tx=0).stop 2024-02-21T23:46:08.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.490+0000 7f7445303700 1 -- 172.21.15.90:0/4216071913 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7440100e00 msgr2=0x7f74401972c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:08.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.490+0000 7f7445303700 1 --2- 172.21.15.90:0/4216071913 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7440100e00 0x7f74401972c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f743400d950 tx=0x7f743400dc60 comp rx=0 tx=0).stop 2024-02-21T23:46:08.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.490+0000 7f7445303700 1 -- 172.21.15.90:0/4216071913 shutdown_connections 2024-02-21T23:46:08.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.490+0000 7f7445303700 1 --2- 172.21.15.90:0/4216071913 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f74280a1120 0x7f74280a35e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:08.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.490+0000 7f7445303700 1 --2- 172.21.15.90:0/4216071913 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f74401017a0 0x7f744019e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:08.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.490+0000 7f7445303700 1 --2- 172.21.15.90:0/4216071913 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f74400ffc10 0x7f7440196d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:08.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.490+0000 7f7445303700 1 --2- 172.21.15.90:0/4216071913 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7440100e00 0x7f74401972c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:08.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.490+0000 7f7445303700 1 -- 172.21.15.90:0/4216071913 >> 172.21.15.90:0/4216071913 conn(0x7f74400fb180 msgr2=0x7f74400fcc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:08.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.490+0000 7f7445303700 1 -- 172.21.15.90:0/4216071913 shutdown_connections 2024-02-21T23:46:08.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.490+0000 7f7445303700 1 -- 172.21.15.90:0/4216071913 wait complete. 2024-02-21T23:46:08.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:08 smithi090 ceph-mon[43385]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi090=a;smithi090=d;smithi090=g;smithi090=j;smithi176=b;smithi176=e;smithi176=h;smithi176=k;smithi178=c;smithi178=f;smithi178=i;smithi178=l", "target": ["mon-mgr", ""]}]: dispatch 2024-02-21T23:46:08.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:08 smithi090 ceph-mon[43385]: Saving service mds.all spec with placement smithi090=a;smithi090=d;smithi090=g;smithi090=j;smithi176=b;smithi176=e;smithi176=h;smithi176=k;smithi178=c;smithi178=f;smithi178=i;smithi178=l;count:12 2024-02-21T23:46:08.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:08 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:08.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:08 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:46:08.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:08 smithi090 ceph-mon[43385]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:08.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:08 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:08.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:08 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:46:08.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:08 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:08.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:08 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-21T23:46:08.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:08 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-21T23:46:08.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:08 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:08.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:08 smithi090 ceph-mon[43385]: Deploying daemon mds.l on smithi178 2024-02-21T23:46:08.486 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:46:08.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.625+0000 7f5e19c25700 1 -- 172.21.15.90:0/432343399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e14101680 msgr2=0x7f5e14107f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:08.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.625+0000 7f5e19c25700 1 --2- 172.21.15.90:0/432343399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e14101680 0x7f5e14107f30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c00b0d0 tx=0x7f5e0c00b3e0 comp rx=0 tx=0).stop 2024-02-21T23:46:08.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.625+0000 7f5e19c25700 1 -- 172.21.15.90:0/432343399 shutdown_connections 2024-02-21T23:46:08.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.625+0000 7f5e19c25700 1 --2- 172.21.15.90:0/432343399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e14101680 0x7f5e14107f30 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:08.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.625+0000 7f5e19c25700 1 --2- 172.21.15.90:0/432343399 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5e140ffaf0 0x7f5e140fff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:08.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.625+0000 7f5e19c25700 1 --2- 172.21.15.90:0/432343399 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5e14100ce0 0x7f5e14101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:08.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.625+0000 7f5e19c25700 1 -- 172.21.15.90:0/432343399 >> 172.21.15.90:0/432343399 conn(0x7f5e140fb060 msgr2=0x7f5e140fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:08.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.625+0000 7f5e19c25700 1 -- 172.21.15.90:0/432343399 shutdown_connections 2024-02-21T23:46:08.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.625+0000 7f5e19c25700 1 -- 172.21.15.90:0/432343399 wait complete. 2024-02-21T23:46:08.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.626+0000 7f5e19c25700 1 Processor -- start 2024-02-21T23:46:08.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.626+0000 7f5e19c25700 1 -- start start 2024-02-21T23:46:08.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.626+0000 7f5e19c25700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5e140ffaf0 0x7f5e14195b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:08.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.626+0000 7f5e19c25700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5e14100ce0 0x7f5e14196080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:08.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.626+0000 7f5e19c25700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e14101680 0x7f5e1419d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:08.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.626+0000 7f5e19c25700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1419d670 con 0x7f5e14101680 2024-02-21T23:46:08.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.626+0000 7f5e19c25700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1419d7b0 con 0x7f5e140ffaf0 2024-02-21T23:46:08.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.626+0000 7f5e19c25700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1419d940 con 0x7f5e14100ce0 2024-02-21T23:46:08.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.627+0000 7f5e13fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e14101680 0x7f5e1419d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:08.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.627+0000 7f5e13fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e14101680 0x7f5e1419d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49448/0 (socket says 172.21.15.90:49448) 2024-02-21T23:46:08.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.627+0000 7f5e13fff700 1 -- 172.21.15.90:0/3842020401 learned_addr learned my addr 172.21.15.90:0/3842020401 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:08.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.627+0000 7f5e12ffd700 1 --2- 172.21.15.90:0/3842020401 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5e14100ce0 0x7f5e14196080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:08.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.627+0000 7f5e137fe700 1 --2- 172.21.15.90:0/3842020401 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5e140ffaf0 0x7f5e14195b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:08.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.627+0000 7f5e13fff700 1 -- 172.21.15.90:0/3842020401 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5e14100ce0 msgr2=0x7f5e14196080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:08.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.627+0000 7f5e13fff700 1 --2- 172.21.15.90:0/3842020401 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5e14100ce0 0x7f5e14196080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:08.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.627+0000 7f5e13fff700 1 -- 172.21.15.90:0/3842020401 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5e140ffaf0 msgr2=0x7f5e14195b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:08.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.627+0000 7f5e13fff700 1 --2- 172.21.15.90:0/3842020401 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5e140ffaf0 0x7f5e14195b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:08.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.627+0000 7f5e13fff700 1 -- 172.21.15.90:0/3842020401 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e0c009d00 con 0x7f5e14101680 2024-02-21T23:46:08.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.627+0000 7f5e13fff700 1 --2- 172.21.15.90:0/3842020401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e14101680 0x7f5e1419d0a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c007b60 tx=0x7f5e0c008f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:08.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.627+0000 7f5e10ff9700 1 -- 172.21.15.90:0/3842020401 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e0c00e040 con 0x7f5e14101680 2024-02-21T23:46:08.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.627+0000 7f5e19c25700 1 -- 172.21.15.90:0/3842020401 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e1419dbc0 con 0x7f5e14101680 2024-02-21T23:46:08.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.627+0000 7f5e10ff9700 1 -- 172.21.15.90:0/3842020401 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5e0c00be30 con 0x7f5e14101680 2024-02-21T23:46:08.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.627+0000 7f5e10ff9700 1 -- 172.21.15.90:0/3842020401 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e0c01bae0 con 0x7f5e14101680 2024-02-21T23:46:08.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.627+0000 7f5e19c25700 1 -- 172.21.15.90:0/3842020401 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e1419e040 con 0x7f5e14101680 2024-02-21T23:46:08.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.628+0000 7f5e19c25700 1 -- 172.21.15.90:0/3842020401 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e14106ea0 con 0x7f5e14101680 2024-02-21T23:46:08.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.629+0000 7f5e10ff9700 1 -- 172.21.15.90:0/3842020401 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7f5e0c01d670 con 0x7f5e14101680 2024-02-21T23:46:08.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.630+0000 7f5e10ff9700 1 --2- 172.21.15.90:0/3842020401 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f5dfc0a11a0 0x7f5dfc0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:08.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.630+0000 7f5e137fe700 1 --2- 172.21.15.90:0/3842020401 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f5dfc0a11a0 0x7f5dfc0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:08.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.630+0000 7f5e10ff9700 1 -- 172.21.15.90:0/3842020401 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f5e0c0ce330 con 0x7f5e14101680 2024-02-21T23:46:08.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.630+0000 7f5e137fe700 1 --2- 172.21.15.90:0/3842020401 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f5dfc0a11a0 0x7f5dfc0a3660 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5e14100b40 tx=0x7f5e0400b5b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:08.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.633+0000 7f5e10ff9700 1 -- 172.21.15.90:0/3842020401 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f5e0c097f50 con 0x7f5e14101680 2024-02-21T23:46:08.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.874+0000 7f5e19c25700 1 -- 172.21.15.90:0/3842020401 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e140679a0 con 0x7f5e14101680 2024-02-21T23:46:08.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.874+0000 7f5e10ff9700 1 -- 172.21.15.90:0/3842020401 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+501 (secure 0 0 0) 0x7f5e0c0bb090 con 0x7f5e14101680 2024-02-21T23:46:08.858 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:46:08.858 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-02-21T23:46:08.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.876+0000 7f5e19c25700 1 -- 172.21.15.90:0/3842020401 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f5dfc0a11a0 msgr2=0x7f5dfc0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:08.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.876+0000 7f5e19c25700 1 --2- 172.21.15.90:0/3842020401 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f5dfc0a11a0 0x7f5dfc0a3660 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5e14100b40 tx=0x7f5e0400b5b0 comp rx=0 tx=0).stop 2024-02-21T23:46:08.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.876+0000 7f5e19c25700 1 -- 172.21.15.90:0/3842020401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e14101680 msgr2=0x7f5e1419d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:08.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.876+0000 7f5e19c25700 1 --2- 172.21.15.90:0/3842020401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e14101680 0x7f5e1419d0a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c007b60 tx=0x7f5e0c008f10 comp rx=0 tx=0).stop 2024-02-21T23:46:08.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.876+0000 7f5e19c25700 1 -- 172.21.15.90:0/3842020401 shutdown_connections 2024-02-21T23:46:08.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.876+0000 7f5e19c25700 1 --2- 172.21.15.90:0/3842020401 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f5dfc0a11a0 0x7f5dfc0a3660 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:08.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.876+0000 7f5e19c25700 1 --2- 172.21.15.90:0/3842020401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e14101680 0x7f5e1419d0a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:08.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.876+0000 7f5e19c25700 1 --2- 172.21.15.90:0/3842020401 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5e140ffaf0 0x7f5e14195b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:08.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.876+0000 7f5e19c25700 1 --2- 172.21.15.90:0/3842020401 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5e14100ce0 0x7f5e14196080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:08.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.876+0000 7f5e19c25700 1 -- 172.21.15.90:0/3842020401 >> 172.21.15.90:0/3842020401 conn(0x7f5e140fb060 msgr2=0x7f5e14103fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:08.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.877+0000 7f5e19c25700 1 -- 172.21.15.90:0/3842020401 shutdown_connections 2024-02-21T23:46:08.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:08.877+0000 7f5e19c25700 1 -- 172.21.15.90:0/3842020401 wait complete. 2024-02-21T23:46:08.861 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 1 2024-02-21T23:46:08.924 INFO:journalctl@ceph.mds.l.smithi178.stdout:Feb 21 23:46:08 smithi178 systemd[1]: Starting Ceph mds.l for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:46:09.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:09 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/2334908623' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-21T23:46:09.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:09 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/4216071913' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-21T23:46:09.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:09 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/3842020401' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-21T23:46:09.333 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:09 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/2334908623' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-21T23:46:09.333 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:09 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/4216071913' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-21T23:46:09.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:09 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/3842020401' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-21T23:46:09.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:09 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/2334908623' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-21T23:46:09.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:09 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/4216071913' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-21T23:46:09.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:09 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/3842020401' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-21T23:46:09.424 INFO:journalctl@ceph.mds.l.smithi178.stdout:Feb 21 23:46:09 smithi178 ceph-mds[84704]: -- 172.21.15.178:0/1920552252 <== mon.1 v2:172.21.15.178:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559f3ce631e0 con 0x559f3ce38800 2024-02-21T23:46:09.424 INFO:journalctl@ceph.mds.l.smithi178.stdout:Feb 21 23:46:09 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-l[84682]: starting mds.l at 2024-02-21T23:46:09.424 INFO:journalctl@ceph.mds.l.smithi178.stdout:Feb 21 23:46:09 smithi178 bash[84667]: c852d5a5eda4cc61a48da2180836b721a9f49147309bc8134c6d911edf9d5850 2024-02-21T23:46:09.424 INFO:journalctl@ceph.mds.l.smithi178.stdout:Feb 21 23:46:09 smithi178 systemd[1]: Started Ceph mds.l for 7ed81218-d112-11ee-95bd-87774f69a715. 2024-02-21T23:46:10.352 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:10 smithi176 ceph-mon[47172]: mds.? [v2:172.21.15.178:6832/1403694104,v1:172.21.15.178:6833/1403694104] up:boot 2024-02-21T23:46:10.353 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:10 smithi176 ceph-mon[47172]: fsmap 1 up:standby 2024-02-21T23:46:10.353 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:10 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-02-21T23:46:10.353 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:10 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:10.353 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:10 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:10.353 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:10 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:10.353 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:10 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-21T23:46:10.353 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:10 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-21T23:46:10.353 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:10 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:10.353 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:10 smithi176 ceph-mon[47172]: Deploying daemon mds.k on smithi176 2024-02-21T23:46:10.353 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:10 smithi176 ceph-mon[47172]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:10.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:10 smithi178 ceph-mon[47547]: mds.? [v2:172.21.15.178:6832/1403694104,v1:172.21.15.178:6833/1403694104] up:boot 2024-02-21T23:46:10.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:10 smithi178 ceph-mon[47547]: fsmap 1 up:standby 2024-02-21T23:46:10.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:10 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-02-21T23:46:10.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:10 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:10.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:10 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:10.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:10 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:10.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:10 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-21T23:46:10.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:10 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-21T23:46:10.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:10 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:10.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:10 smithi178 ceph-mon[47547]: Deploying daemon mds.k on smithi176 2024-02-21T23:46:10.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:10 smithi178 ceph-mon[47547]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:10.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:10 smithi090 ceph-mon[43385]: mds.? [v2:172.21.15.178:6832/1403694104,v1:172.21.15.178:6833/1403694104] up:boot 2024-02-21T23:46:10.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:10 smithi090 ceph-mon[43385]: fsmap 1 up:standby 2024-02-21T23:46:10.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:10 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-02-21T23:46:10.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:10 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:10.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:10 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:10.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:10 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:10.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:10 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-21T23:46:10.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:10 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-21T23:46:10.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:10 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:10.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:10 smithi090 ceph-mon[43385]: Deploying daemon mds.k on smithi176 2024-02-21T23:46:10.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:10 smithi090 ceph-mon[43385]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:10.830 INFO:journalctl@ceph.mds.k.smithi176.stdout:Feb 21 23:46:10 smithi176 systemd[1]: Starting Ceph mds.k for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:46:10.872 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:46:11.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.025+0000 7fb765bd6700 1 -- 172.21.15.90:0/3150037496 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb760101e20 msgr2=0x7fb760102280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:11.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.025+0000 7fb765bd6700 1 --2- 172.21.15.90:0/3150037496 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb760101e20 0x7fb760102280 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb7540099c0 tx=0x7fb754009cd0 comp rx=0 tx=0).stop 2024-02-21T23:46:11.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.025+0000 7fb765bd6700 1 -- 172.21.15.90:0/3150037496 shutdown_connections 2024-02-21T23:46:11.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.025+0000 7fb765bd6700 1 --2- 172.21.15.90:0/3150037496 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7601027c0 0x7fb760109070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:11.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.025+0000 7fb765bd6700 1 --2- 172.21.15.90:0/3150037496 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb760100c30 0x7fb760101050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:11.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.025+0000 7fb765bd6700 1 --2- 172.21.15.90:0/3150037496 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb760101e20 0x7fb760102280 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:11.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.025+0000 7fb765bd6700 1 -- 172.21.15.90:0/3150037496 >> 172.21.15.90:0/3150037496 conn(0x7fb7600fc1a0 msgr2=0x7fb7600fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:11.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.025+0000 7fb765bd6700 1 -- 172.21.15.90:0/3150037496 shutdown_connections 2024-02-21T23:46:11.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.025+0000 7fb765bd6700 1 -- 172.21.15.90:0/3150037496 wait complete. 2024-02-21T23:46:11.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.026+0000 7fb765bd6700 1 Processor -- start 2024-02-21T23:46:11.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.026+0000 7fb765bd6700 1 -- start start 2024-02-21T23:46:11.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.026+0000 7fb765bd6700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb760100c30 0x7fb760196c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:11.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.026+0000 7fb765bd6700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb760101e20 0x7fb7601971d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:11.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.026+0000 7fb765bd6700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb7601027c0 0x7fb76019e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:11.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.026+0000 7fb765bd6700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76019e780 con 0x7fb7601027c0 2024-02-21T23:46:11.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.026+0000 7fb765bd6700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76019e8f0 con 0x7fb760101e20 2024-02-21T23:46:11.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.026+0000 7fb765bd6700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76019eab0 con 0x7fb760100c30 2024-02-21T23:46:11.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.026+0000 7fb75ffff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb7601027c0 0x7fb76019e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:11.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.026+0000 7fb75effd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb760101e20 0x7fb7601971d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:11.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.026+0000 7fb75effd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb760101e20 0x7fb7601971d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.90:40336/0 (socket says 172.21.15.90:40336) 2024-02-21T23:46:11.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.026+0000 7fb75ffff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb7601027c0 0x7fb76019e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49462/0 (socket says 172.21.15.90:49462) 2024-02-21T23:46:11.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.026+0000 7fb75effd700 1 -- 172.21.15.90:0/1626581124 learned_addr learned my addr 172.21.15.90:0/1626581124 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:11.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.026+0000 7fb75f7fe700 1 --2- 172.21.15.90:0/1626581124 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb760100c30 0x7fb760196c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:11.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.027+0000 7fb75effd700 1 -- 172.21.15.90:0/1626581124 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb760100c30 msgr2=0x7fb760196c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:11.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.027+0000 7fb75effd700 1 --2- 172.21.15.90:0/1626581124 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb760100c30 0x7fb760196c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:11.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.027+0000 7fb75effd700 1 -- 172.21.15.90:0/1626581124 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb7601027c0 msgr2=0x7fb76019e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:11.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.027+0000 7fb75effd700 1 --2- 172.21.15.90:0/1626581124 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb7601027c0 0x7fb76019e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:11.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.027+0000 7fb75effd700 1 -- 172.21.15.90:0/1626581124 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7540095f0 con 0x7fb760101e20 2024-02-21T23:46:11.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.027+0000 7fb75ffff700 1 --2- 172.21.15.90:0/1626581124 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb7601027c0 0x7fb76019e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:46:11.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.027+0000 7fb75effd700 1 --2- 172.21.15.90:0/1626581124 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb760101e20 0x7fb7601971d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb754006010 tx=0x7fb754004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:11.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.027+0000 7fb75cff9700 1 -- 172.21.15.90:0/1626581124 <== mon.2 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb75401d070 con 0x7fb760101e20 2024-02-21T23:46:11.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.027+0000 7fb75cff9700 1 -- 172.21.15.90:0/1626581124 <== mon.2 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb754003f90 con 0x7fb760101e20 2024-02-21T23:46:11.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.027+0000 7fb765bd6700 1 -- 172.21.15.90:0/1626581124 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb76019ed30 con 0x7fb760101e20 2024-02-21T23:46:11.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.027+0000 7fb75cff9700 1 -- 172.21.15.90:0/1626581124 <== mon.2 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb75400fd70 con 0x7fb760101e20 2024-02-21T23:46:11.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.027+0000 7fb765bd6700 1 -- 172.21.15.90:0/1626581124 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb76019f150 con 0x7fb760101e20 2024-02-21T23:46:11.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.028+0000 7fb765bd6700 1 -- 172.21.15.90:0/1626581124 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb760108150 con 0x7fb760101e20 2024-02-21T23:46:11.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.029+0000 7fb75cff9700 1 -- 172.21.15.90:0/1626581124 <== mon.2 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7fb75400f460 con 0x7fb760101e20 2024-02-21T23:46:11.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.030+0000 7fb75cff9700 1 --2- 172.21.15.90:0/1626581124 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fb7480a1120 0x7fb7480a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:11.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.030+0000 7fb75f7fe700 1 --2- 172.21.15.90:0/1626581124 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fb7480a1120 0x7fb7480a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:11.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.030+0000 7fb75cff9700 1 -- 172.21.15.90:0/1626581124 <== mon.2 v2:172.21.15.176:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb75409b380 con 0x7fb760101e20 2024-02-21T23:46:11.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.030+0000 7fb75f7fe700 1 --2- 172.21.15.90:0/1626581124 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fb7480a1120 0x7fb7480a35e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb750006fd0 tx=0x7fb750008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:11.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.035+0000 7fb75cff9700 1 -- 172.21.15.90:0/1626581124 <== mon.2 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fb754096fc0 con 0x7fb760101e20 2024-02-21T23:46:11.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.289+0000 7fb765bd6700 1 -- 172.21.15.90:0/1626581124 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7600679a0 con 0x7fb760101e20 2024-02-21T23:46:11.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.290+0000 7fb75cff9700 1 -- 172.21.15.90:0/1626581124 <== mon.2 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1260 (secure 0 0 0) 0x7fb75409a5e0 con 0x7fb760101e20 2024-02-21T23:46:11.274 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:46:11.274 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1403694104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1403694104},{"type":"v1","addr":"172.21.15.178:6833","nonce":1403694104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2024-02-21T23:46:11.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.292+0000 7fb765bd6700 1 -- 172.21.15.90:0/1626581124 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fb7480a1120 msgr2=0x7fb7480a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:11.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.292+0000 7fb765bd6700 1 --2- 172.21.15.90:0/1626581124 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fb7480a1120 0x7fb7480a35e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb750006fd0 tx=0x7fb750008040 comp rx=0 tx=0).stop 2024-02-21T23:46:11.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.292+0000 7fb765bd6700 1 -- 172.21.15.90:0/1626581124 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb760101e20 msgr2=0x7fb7601971d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:11.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.292+0000 7fb765bd6700 1 --2- 172.21.15.90:0/1626581124 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb760101e20 0x7fb7601971d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb754006010 tx=0x7fb754004730 comp rx=0 tx=0).stop 2024-02-21T23:46:11.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.292+0000 7fb765bd6700 1 -- 172.21.15.90:0/1626581124 shutdown_connections 2024-02-21T23:46:11.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.292+0000 7fb765bd6700 1 --2- 172.21.15.90:0/1626581124 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fb7480a1120 0x7fb7480a35e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:11.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.292+0000 7fb765bd6700 1 --2- 172.21.15.90:0/1626581124 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb7601027c0 0x7fb76019e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:11.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.292+0000 7fb765bd6700 1 --2- 172.21.15.90:0/1626581124 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb760100c30 0x7fb760196c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:11.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.292+0000 7fb765bd6700 1 --2- 172.21.15.90:0/1626581124 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb760101e20 0x7fb7601971d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:11.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.293+0000 7fb765bd6700 1 -- 172.21.15.90:0/1626581124 >> 172.21.15.90:0/1626581124 conn(0x7fb7600fc1a0 msgr2=0x7fb760105290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:11.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.293+0000 7fb765bd6700 1 -- 172.21.15.90:0/1626581124 shutdown_connections 2024-02-21T23:46:11.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:11.293+0000 7fb765bd6700 1 -- 172.21.15.90:0/1626581124 wait complete. 2024-02-21T23:46:11.278 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 2 2024-02-21T23:46:11.333 INFO:journalctl@ceph.mds.k.smithi176.stdout:Feb 21 23:46:10 smithi176 ceph-mds[82615]: -- 172.21.15.176:0/346565481 <== mon.2 v2:172.21.15.176:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e2769c91e0 con 0x55e27699e800 2024-02-21T23:46:11.334 INFO:journalctl@ceph.mds.k.smithi176.stdout:Feb 21 23:46:10 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-k[82592]: starting mds.k at 2024-02-21T23:46:11.334 INFO:journalctl@ceph.mds.k.smithi176.stdout:Feb 21 23:46:10 smithi176 bash[82577]: 6fe45890efc5a39d3d3d8d8445568c2e824643da911cb4211beedd5f4e9b5f91 2024-02-21T23:46:11.334 INFO:journalctl@ceph.mds.k.smithi176.stdout:Feb 21 23:46:10 smithi176 systemd[1]: Started Ceph mds.k for 7ed81218-d112-11ee-95bd-87774f69a715. 2024-02-21T23:46:12.217 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:12 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:12.217 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:12 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:12.217 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:12 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:12.217 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:12 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-21T23:46:12.217 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:12 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-21T23:46:12.217 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:12 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:12.218 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:12 smithi090 ceph-mon[43385]: Deploying daemon mds.j on smithi090 2024-02-21T23:46:12.218 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:12 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/1626581124' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-21T23:46:12.218 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:12 smithi090 ceph-mon[43385]: mds.? [v2:172.21.15.176:6832/2956287083,v1:172.21.15.176:6833/2956287083] up:boot 2024-02-21T23:46:12.218 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:12 smithi090 ceph-mon[43385]: fsmap 2 up:standby 2024-02-21T23:46:12.218 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:12 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-02-21T23:46:12.218 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:12 smithi090 ceph-mon[43385]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:12.333 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:12 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:12.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:12 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:12.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:12 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:12.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:12 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-21T23:46:12.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:12 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-21T23:46:12.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:12 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:12.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:12 smithi176 ceph-mon[47172]: Deploying daemon mds.j on smithi090 2024-02-21T23:46:12.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:12 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/1626581124' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-21T23:46:12.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:12 smithi176 ceph-mon[47172]: mds.? [v2:172.21.15.176:6832/2956287083,v1:172.21.15.176:6833/2956287083] up:boot 2024-02-21T23:46:12.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:12 smithi176 ceph-mon[47172]: fsmap 2 up:standby 2024-02-21T23:46:12.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:12 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-02-21T23:46:12.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:12 smithi176 ceph-mon[47172]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:12.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:12 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:12.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:12 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:12.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:12 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:12.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:12 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-21T23:46:12.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:12 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-21T23:46:12.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:12 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:12.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:12 smithi178 ceph-mon[47547]: Deploying daemon mds.j on smithi090 2024-02-21T23:46:12.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:12 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/1626581124' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-21T23:46:12.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:12 smithi178 ceph-mon[47547]: mds.? [v2:172.21.15.176:6832/2956287083,v1:172.21.15.176:6833/2956287083] up:boot 2024-02-21T23:46:12.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:12 smithi178 ceph-mon[47547]: fsmap 2 up:standby 2024-02-21T23:46:12.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:12 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-02-21T23:46:12.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:12 smithi178 ceph-mon[47547]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:12.477 INFO:journalctl@ceph.mds.j.smithi090.stdout:Feb 21 23:46:12 smithi090 systemd[1]: Starting Ceph mds.j for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:46:12.977 INFO:journalctl@ceph.mds.j.smithi090.stdout:Feb 21 23:46:12 smithi090 ceph-mds[93887]: -- 172.21.15.90:0/2219618415 <== mon.0 v2:172.21.15.90:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556ef16a71e0 con 0x556ef167c800 2024-02-21T23:46:12.977 INFO:journalctl@ceph.mds.j.smithi090.stdout:Feb 21 23:46:12 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-j[93864]: starting mds.j at 2024-02-21T23:46:12.977 INFO:journalctl@ceph.mds.j.smithi090.stdout:Feb 21 23:46:12 smithi090 bash[93850]: 5a4968c7fdaa3b8f5baac0259f63d2ac803084edaf0cdf29ba3125b062c13b32 2024-02-21T23:46:12.977 INFO:journalctl@ceph.mds.j.smithi090.stdout:Feb 21 23:46:12 smithi090 systemd[1]: Started Ceph mds.j for 7ed81218-d112-11ee-95bd-87774f69a715. 2024-02-21T23:46:13.294 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:46:13.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.442+0000 7f4e061a1700 1 -- 172.21.15.90:0/889494418 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e00101150 msgr2=0x7f4e00101570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:13.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.442+0000 7f4e061a1700 1 --2- 172.21.15.90:0/889494418 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e00101150 0x7f4e00101570 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc009990 tx=0x7f4dfc009ca0 comp rx=0 tx=0).stop 2024-02-21T23:46:13.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.442+0000 7f4e061a1700 1 -- 172.21.15.90:0/889494418 shutdown_connections 2024-02-21T23:46:13.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.442+0000 7f4e061a1700 1 --2- 172.21.15.90:0/889494418 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e001006d0 0x7f4e00107f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:13.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.442+0000 7f4e061a1700 1 --2- 172.21.15.90:0/889494418 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e00101150 0x7f4e00101570 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:13.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.442+0000 7f4e061a1700 1 --2- 172.21.15.90:0/889494418 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4e000ffd10 0x7f4e00100190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:13.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.442+0000 7f4e061a1700 1 -- 172.21.15.90:0/889494418 >> 172.21.15.90:0/889494418 conn(0x7f4e000fa7c0 msgr2=0x7f4e000fcc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:13.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.442+0000 7f4e061a1700 1 -- 172.21.15.90:0/889494418 shutdown_connections 2024-02-21T23:46:13.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.442+0000 7f4e061a1700 1 -- 172.21.15.90:0/889494418 wait complete. 2024-02-21T23:46:13.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.443+0000 7f4e061a1700 1 Processor -- start 2024-02-21T23:46:13.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.443+0000 7f4e061a1700 1 -- start start 2024-02-21T23:46:13.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.443+0000 7f4e061a1700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4e000ffd10 0x7f4e00195ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:13.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.443+0000 7f4e061a1700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e001006d0 0x7f4e00195ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:13.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.443+0000 7f4e061a1700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e00101150 0x7f4e0019d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:13.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.443+0000 7f4e061a1700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0019d5e0 con 0x7f4e001006d0 2024-02-21T23:46:13.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.443+0000 7f4e061a1700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0019d720 con 0x7f4e00101150 2024-02-21T23:46:13.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.443+0000 7f4e061a1700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0019d8b0 con 0x7f4e000ffd10 2024-02-21T23:46:13.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.443+0000 7f4e0499e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e001006d0 0x7f4e00195ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:13.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.443+0000 7f4e0499e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e001006d0 0x7f4e00195ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49492/0 (socket says 172.21.15.90:49492) 2024-02-21T23:46:13.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.443+0000 7f4e0499e700 1 -- 172.21.15.90:0/78936338 learned_addr learned my addr 172.21.15.90:0/78936338 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:13.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.443+0000 7f4e0519f700 1 --2- 172.21.15.90:0/78936338 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4e000ffd10 0x7f4e00195ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:13.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.443+0000 7f4e059a0700 1 --2- 172.21.15.90:0/78936338 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e00101150 0x7f4e0019d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:13.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.444+0000 7f4e0499e700 1 -- 172.21.15.90:0/78936338 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4e000ffd10 msgr2=0x7f4e00195ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:13.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.444+0000 7f4e0499e700 1 --2- 172.21.15.90:0/78936338 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4e000ffd10 0x7f4e00195ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:13.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.444+0000 7f4e0499e700 1 -- 172.21.15.90:0/78936338 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e00101150 msgr2=0x7f4e0019d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:13.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.444+0000 7f4e0499e700 1 --2- 172.21.15.90:0/78936338 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e00101150 0x7f4e0019d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:13.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.444+0000 7f4e0499e700 1 -- 172.21.15.90:0/78936338 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dfc0095f0 con 0x7f4e001006d0 2024-02-21T23:46:13.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.444+0000 7f4e059a0700 1 --2- 172.21.15.90:0/78936338 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e00101150 0x7f4e0019d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:46:13.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.444+0000 7f4e0499e700 1 --2- 172.21.15.90:0/78936338 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e001006d0 0x7f4e00195ff0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4df000d950 tx=0x7f4df000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:13.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.444+0000 7f4df67fc700 1 -- 172.21.15.90:0/78936338 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df0009940 con 0x7f4e001006d0 2024-02-21T23:46:13.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.444+0000 7f4df67fc700 1 -- 172.21.15.90:0/78936338 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4df0010460 con 0x7f4e001006d0 2024-02-21T23:46:13.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.444+0000 7f4e061a1700 1 -- 172.21.15.90:0/78936338 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e0019db90 con 0x7f4e001006d0 2024-02-21T23:46:13.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.444+0000 7f4df67fc700 1 -- 172.21.15.90:0/78936338 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df000f620 con 0x7f4e001006d0 2024-02-21T23:46:13.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.444+0000 7f4e061a1700 1 -- 172.21.15.90:0/78936338 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e0019e0a0 con 0x7f4e001006d0 2024-02-21T23:46:13.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.445+0000 7f4e061a1700 1 -- 172.21.15.90:0/78936338 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e00106fb0 con 0x7f4e001006d0 2024-02-21T23:46:13.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.446+0000 7f4df67fc700 1 -- 172.21.15.90:0/78936338 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7f4df00108d0 con 0x7f4e001006d0 2024-02-21T23:46:13.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.446+0000 7f4df67fc700 1 --2- 172.21.15.90:0/78936338 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f4dec0a1120 0x7f4dec0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:13.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.447+0000 7f4df67fc700 1 -- 172.21.15.90:0/78936338 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4df00cc290 con 0x7f4e001006d0 2024-02-21T23:46:13.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.450+0000 7f4e0519f700 1 --2- 172.21.15.90:0/78936338 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f4dec0a1120 0x7f4dec0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:13.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.450+0000 7f4df67fc700 1 -- 172.21.15.90:0/78936338 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f4df0095ec0 con 0x7f4e001006d0 2024-02-21T23:46:13.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.450+0000 7f4e0519f700 1 --2- 172.21.15.90:0/78936338 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f4dec0a1120 0x7f4dec0a35e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc00b5c0 tx=0x7f4dfc003790 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:13.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.704+0000 7f4e061a1700 1 -- 172.21.15.90:0/78936338 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e000670a0 con 0x7f4e001006d0 2024-02-21T23:46:13.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.705+0000 7f4df67fc700 1 -- 172.21.15.90:0/78936338 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2777 (secure 0 0 0) 0x7f4df0016af0 con 0x7f4e001006d0 2024-02-21T23:46:13.689 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:46:13.689 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/1591404300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1591404300},{"type":"v1","addr":"172.21.15.90:6835","nonce":1591404300}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1403694104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1403694104},{"type":"v1","addr":"172.21.15.178:6833","nonce":1403694104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/2956287083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":2956287083},{"type":"v1","addr":"172.21.15.176:6833","nonce":2956287083}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2024-02-21T23:46:13.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.707+0000 7f4e061a1700 1 -- 172.21.15.90:0/78936338 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f4dec0a1120 msgr2=0x7f4dec0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:13.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.707+0000 7f4e061a1700 1 --2- 172.21.15.90:0/78936338 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f4dec0a1120 0x7f4dec0a35e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc00b5c0 tx=0x7f4dfc003790 comp rx=0 tx=0).stop 2024-02-21T23:46:13.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.707+0000 7f4e061a1700 1 -- 172.21.15.90:0/78936338 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e001006d0 msgr2=0x7f4e00195ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:13.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.707+0000 7f4e061a1700 1 --2- 172.21.15.90:0/78936338 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e001006d0 0x7f4e00195ff0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4df000d950 tx=0x7f4df000dc60 comp rx=0 tx=0).stop 2024-02-21T23:46:13.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.707+0000 7f4e061a1700 1 -- 172.21.15.90:0/78936338 shutdown_connections 2024-02-21T23:46:13.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.707+0000 7f4e061a1700 1 --2- 172.21.15.90:0/78936338 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f4dec0a1120 0x7f4dec0a35e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:13.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.707+0000 7f4e061a1700 1 --2- 172.21.15.90:0/78936338 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e00101150 0x7f4e0019d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:13.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.707+0000 7f4e061a1700 1 --2- 172.21.15.90:0/78936338 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4e000ffd10 0x7f4e00195ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:13.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.707+0000 7f4e061a1700 1 --2- 172.21.15.90:0/78936338 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e001006d0 0x7f4e00195ff0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:13.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.707+0000 7f4e061a1700 1 -- 172.21.15.90:0/78936338 >> 172.21.15.90:0/78936338 conn(0x7f4e000fa7c0 msgr2=0x7f4e001040f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:13.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.707+0000 7f4e061a1700 1 -- 172.21.15.90:0/78936338 shutdown_connections 2024-02-21T23:46:13.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:13.707+0000 7f4e061a1700 1 -- 172.21.15.90:0/78936338 wait complete. 2024-02-21T23:46:13.693 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 4 2024-02-21T23:46:13.879 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:13 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:13.879 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:13 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:13.879 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:13 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:13.879 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:13 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-21T23:46:13.879 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:13 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-21T23:46:13.879 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:13 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:13.879 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:13 smithi090 ceph-mon[43385]: Deploying daemon mds.d on smithi090 2024-02-21T23:46:13.879 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:13 smithi090 ceph-mon[43385]: mds.? [v2:172.21.15.90:6834/1591404300,v1:172.21.15.90:6835/1591404300] up:boot 2024-02-21T23:46:13.879 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:13 smithi090 ceph-mon[43385]: fsmap 3 up:standby 2024-02-21T23:46:13.879 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:13 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-02-21T23:46:13.879 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:13 smithi090 ceph-mon[43385]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:13.879 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:13 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/78936338' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-21T23:46:14.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:13 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:14.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:13 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:14.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:13 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:14.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:13 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-21T23:46:14.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:13 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-21T23:46:14.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:13 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:14.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:13 smithi176 ceph-mon[47172]: Deploying daemon mds.d on smithi090 2024-02-21T23:46:14.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:13 smithi176 ceph-mon[47172]: mds.? [v2:172.21.15.90:6834/1591404300,v1:172.21.15.90:6835/1591404300] up:boot 2024-02-21T23:46:14.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:13 smithi176 ceph-mon[47172]: fsmap 3 up:standby 2024-02-21T23:46:14.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:13 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-02-21T23:46:14.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:13 smithi176 ceph-mon[47172]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:14.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:13 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/78936338' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-21T23:46:14.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:13 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:14.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:13 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:14.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:13 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:14.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:13 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-21T23:46:14.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:13 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-21T23:46:14.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:13 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:14.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:13 smithi178 ceph-mon[47547]: Deploying daemon mds.d on smithi090 2024-02-21T23:46:14.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:13 smithi178 ceph-mon[47547]: mds.? [v2:172.21.15.90:6834/1591404300,v1:172.21.15.90:6835/1591404300] up:boot 2024-02-21T23:46:14.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:13 smithi178 ceph-mon[47547]: fsmap 3 up:standby 2024-02-21T23:46:14.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:13 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-02-21T23:46:14.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:13 smithi178 ceph-mon[47547]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:14.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:13 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/78936338' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-21T23:46:14.227 INFO:journalctl@ceph.mds.d.smithi090.stdout:Feb 21 23:46:14 smithi090 systemd[1]: Starting Ceph mds.d for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:46:14.977 INFO:journalctl@ceph.mds.d.smithi090.stdout:Feb 21 23:46:14 smithi090 ceph-mds[94273]: -- 172.21.15.90:0/948312750 <== mon.0 v2:172.21.15.90:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560a2222d1e0 con 0x560a22202c00 2024-02-21T23:46:14.977 INFO:journalctl@ceph.mds.d.smithi090.stdout:Feb 21 23:46:14 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-d[94250]: starting mds.d at 2024-02-21T23:46:14.977 INFO:journalctl@ceph.mds.d.smithi090.stdout:Feb 21 23:46:14 smithi090 bash[94235]: ac49771c44f76cb26fbbfc0c7f1dcbe8b67b479b794201bb1eea11ed0d13a9b2 2024-02-21T23:46:14.977 INFO:journalctl@ceph.mds.d.smithi090.stdout:Feb 21 23:46:14 smithi090 systemd[1]: Started Ceph mds.d for 7ed81218-d112-11ee-95bd-87774f69a715. 2024-02-21T23:46:15.697 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:15 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:15.697 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:15 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:15.697 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:15 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:15.697 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:15 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-21T23:46:15.697 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:15 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-21T23:46:15.697 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:15 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:15.698 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:15 smithi176 ceph-mon[47172]: Deploying daemon mds.e on smithi176 2024-02-21T23:46:15.698 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:15 smithi176 ceph-mon[47172]: mds.? [v2:172.21.15.90:6836/1441666263,v1:172.21.15.90:6837/1441666263] up:boot 2024-02-21T23:46:15.698 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:15 smithi176 ceph-mon[47172]: fsmap 4 up:standby 2024-02-21T23:46:15.698 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:15 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-02-21T23:46:15.704 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:46:15.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.862+0000 7f286a88f700 1 -- 172.21.15.90:0/3619529124 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2864101df0 msgr2=0x7f2864102250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:15.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.862+0000 7f286a88f700 1 --2- 172.21.15.90:0/3619529124 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2864101df0 0x7f2864102250 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f28580099c0 tx=0x7f2858009cd0 comp rx=0 tx=0).stop 2024-02-21T23:46:15.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.862+0000 7f286a88f700 1 -- 172.21.15.90:0/3619529124 shutdown_connections 2024-02-21T23:46:15.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.862+0000 7f286a88f700 1 --2- 172.21.15.90:0/3619529124 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2864102790 0x7f2864109040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:15.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.862+0000 7f286a88f700 1 --2- 172.21.15.90:0/3619529124 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2864100c00 0x7f2864101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:15.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.862+0000 7f286a88f700 1 --2- 172.21.15.90:0/3619529124 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2864101df0 0x7f2864102250 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:15.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.862+0000 7f286a88f700 1 -- 172.21.15.90:0/3619529124 >> 172.21.15.90:0/3619529124 conn(0x7f28640fc170 msgr2=0x7f28640fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:15.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.862+0000 7f286a88f700 1 -- 172.21.15.90:0/3619529124 shutdown_connections 2024-02-21T23:46:15.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.862+0000 7f286a88f700 1 -- 172.21.15.90:0/3619529124 wait complete. 2024-02-21T23:46:15.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.863+0000 7f286a88f700 1 Processor -- start 2024-02-21T23:46:15.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.863+0000 7f286a88f700 1 -- start start 2024-02-21T23:46:15.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.863+0000 7f286a88f700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2864100c00 0x7f2864196ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:15.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.864+0000 7f286a88f700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2864101df0 0x7f28641970e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:15.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.864+0000 7f286a88f700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2864102790 0x7f286419e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:15.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.864+0000 7f286a88f700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286419e6d0 con 0x7f2864102790 2024-02-21T23:46:15.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.864+0000 7f286a88f700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286419e810 con 0x7f2864101df0 2024-02-21T23:46:15.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.864+0000 7f286a88f700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286419e9a0 con 0x7f2864100c00 2024-02-21T23:46:15.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.864+0000 7f2868e2c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2864102790 0x7f286419e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:15.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.864+0000 7f2868e2c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2864102790 0x7f286419e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49534/0 (socket says 172.21.15.90:49534) 2024-02-21T23:46:15.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.864+0000 7f2868e2c700 1 -- 172.21.15.90:0/1910151909 learned_addr learned my addr 172.21.15.90:0/1910151909 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:15.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.864+0000 7f2863fff700 1 --2- 172.21.15.90:0/1910151909 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2864100c00 0x7f2864196ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:15.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.864+0000 7f28637fe700 1 --2- 172.21.15.90:0/1910151909 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2864101df0 0x7f28641970e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:15.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.864+0000 7f2868e2c700 1 -- 172.21.15.90:0/1910151909 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2864100c00 msgr2=0x7f2864196ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:15.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.864+0000 7f2868e2c700 1 --2- 172.21.15.90:0/1910151909 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2864100c00 0x7f2864196ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:15.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.864+0000 7f2868e2c700 1 -- 172.21.15.90:0/1910151909 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2864101df0 msgr2=0x7f28641970e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:15.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.864+0000 7f2868e2c700 1 --2- 172.21.15.90:0/1910151909 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2864101df0 0x7f28641970e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:15.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.864+0000 7f2868e2c700 1 -- 172.21.15.90:0/1910151909 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28580095f0 con 0x7f2864102790 2024-02-21T23:46:15.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.864+0000 7f2868e2c700 1 --2- 172.21.15.90:0/1910151909 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2864102790 0x7f286419e100 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f285c007ee0 tx=0x7f285c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:15.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.864+0000 7f28617fa700 1 -- 172.21.15.90:0/1910151909 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285c017070 con 0x7f2864102790 2024-02-21T23:46:15.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.864+0000 7f28617fa700 1 -- 172.21.15.90:0/1910151909 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f285c00f040 con 0x7f2864102790 2024-02-21T23:46:15.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.864+0000 7f286a88f700 1 -- 172.21.15.90:0/1910151909 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f286419ec80 con 0x7f2864102790 2024-02-21T23:46:15.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.864+0000 7f28617fa700 1 -- 172.21.15.90:0/1910151909 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285c005580 con 0x7f2864102790 2024-02-21T23:46:15.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.864+0000 7f286a88f700 1 -- 172.21.15.90:0/1910151909 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f286419f140 con 0x7f2864102790 2024-02-21T23:46:15.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.865+0000 7f286a88f700 1 -- 172.21.15.90:0/1910151909 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2864108080 con 0x7f2864102790 2024-02-21T23:46:15.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.866+0000 7f28617fa700 1 -- 172.21.15.90:0/1910151909 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7f285c00db20 con 0x7f2864102790 2024-02-21T23:46:15.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.867+0000 7f28617fa700 1 --2- 172.21.15.90:0/1910151909 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f284c0a11a0 0x7f284c0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:15.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.867+0000 7f2863fff700 1 --2- 172.21.15.90:0/1910151909 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f284c0a11a0 0x7f284c0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:15.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.867+0000 7f28617fa700 1 -- 172.21.15.90:0/1910151909 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f285c0cc2b0 con 0x7f2864102790 2024-02-21T23:46:15.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.867+0000 7f2863fff700 1 --2- 172.21.15.90:0/1910151909 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f284c0a11a0 0x7f284c0a3660 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2864101c50 tx=0x7f2854009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:15.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:15.870+0000 7f28617fa700 1 -- 172.21.15.90:0/1910151909 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f285c095e40 con 0x7f2864102790 2024-02-21T23:46:15.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:15 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:15.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:15 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:15.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:15 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:15.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:15 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-21T23:46:15.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:15 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-21T23:46:15.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:15 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:15.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:15 smithi178 ceph-mon[47547]: Deploying daemon mds.e on smithi176 2024-02-21T23:46:15.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:15 smithi178 ceph-mon[47547]: mds.? [v2:172.21.15.90:6836/1441666263,v1:172.21.15.90:6837/1441666263] up:boot 2024-02-21T23:46:15.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:15 smithi178 ceph-mon[47547]: fsmap 4 up:standby 2024-02-21T23:46:15.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:15 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-02-21T23:46:15.947 INFO:journalctl@ceph.mds.e.smithi176.stdout:Feb 21 23:46:15 smithi176 systemd[1]: Starting Ceph mds.e for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:46:15.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:15 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:15.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:15 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:15.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:15 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:15.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:15 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-21T23:46:15.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:15 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-21T23:46:15.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:15 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:15.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:15 smithi090 ceph-mon[43385]: Deploying daemon mds.e on smithi176 2024-02-21T23:46:15.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:15 smithi090 ceph-mon[43385]: mds.? [v2:172.21.15.90:6836/1441666263,v1:172.21.15.90:6837/1441666263] up:boot 2024-02-21T23:46:15.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:15 smithi090 ceph-mon[43385]: fsmap 4 up:standby 2024-02-21T23:46:15.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:15 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-02-21T23:46:16.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:16.110+0000 7f286a88f700 1 -- 172.21.15.90:0/1910151909 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2864067df0 con 0x7f2864102790 2024-02-21T23:46:16.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:16.110+0000 7f28617fa700 1 -- 172.21.15.90:0/1910151909 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3534 (secure 0 0 0) 0x7f285c099460 con 0x7f2864102790 2024-02-21T23:46:16.095 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:46:16.095 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/1591404300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1591404300},{"type":"v1","addr":"172.21.15.90:6835","nonce":1591404300}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/1441666263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":1441666263},{"type":"v1","addr":"172.21.15.90:6837","nonce":1441666263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1403694104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1403694104},{"type":"v1","addr":"172.21.15.178:6833","nonce":1403694104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/2956287083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":2956287083},{"type":"v1","addr":"172.21.15.176:6833","nonce":2956287083}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2024-02-21T23:46:16.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:16.112+0000 7f286a88f700 1 -- 172.21.15.90:0/1910151909 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f284c0a11a0 msgr2=0x7f284c0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:16.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:16.112+0000 7f286a88f700 1 --2- 172.21.15.90:0/1910151909 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f284c0a11a0 0x7f284c0a3660 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2864101c50 tx=0x7f2854009260 comp rx=0 tx=0).stop 2024-02-21T23:46:16.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:16.112+0000 7f286a88f700 1 -- 172.21.15.90:0/1910151909 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2864102790 msgr2=0x7f286419e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:16.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:16.112+0000 7f286a88f700 1 --2- 172.21.15.90:0/1910151909 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2864102790 0x7f286419e100 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f285c007ee0 tx=0x7f285c00d3b0 comp rx=0 tx=0).stop 2024-02-21T23:46:16.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:16.113+0000 7f286a88f700 1 -- 172.21.15.90:0/1910151909 shutdown_connections 2024-02-21T23:46:16.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:16.113+0000 7f286a88f700 1 --2- 172.21.15.90:0/1910151909 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f284c0a11a0 0x7f284c0a3660 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:16.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:16.113+0000 7f286a88f700 1 --2- 172.21.15.90:0/1910151909 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2864102790 0x7f286419e100 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:16.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:16.113+0000 7f286a88f700 1 --2- 172.21.15.90:0/1910151909 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2864100c00 0x7f2864196ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:16.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:16.113+0000 7f286a88f700 1 --2- 172.21.15.90:0/1910151909 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2864101df0 0x7f28641970e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:16.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:16.113+0000 7f286a88f700 1 -- 172.21.15.90:0/1910151909 >> 172.21.15.90:0/1910151909 conn(0x7f28640fc170 msgr2=0x7f28641051c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:16.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:16.113+0000 7f286a88f700 1 -- 172.21.15.90:0/1910151909 shutdown_connections 2024-02-21T23:46:16.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:16.113+0000 7f286a88f700 1 -- 172.21.15.90:0/1910151909 wait complete. 2024-02-21T23:46:16.098 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 5 2024-02-21T23:46:16.584 INFO:journalctl@ceph.mds.e.smithi176.stdout:Feb 21 23:46:16 smithi176 ceph-mds[82958]: -- 172.21.15.176:0/1432541931 <== mon.2 v2:172.21.15.176:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a730d5f1e0 con 0x55a730d34800 2024-02-21T23:46:16.584 INFO:journalctl@ceph.mds.e.smithi176.stdout:Feb 21 23:46:16 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-e[82935]: starting mds.e at 2024-02-21T23:46:16.584 INFO:journalctl@ceph.mds.e.smithi176.stdout:Feb 21 23:46:16 smithi176 bash[82920]: 9a298e9ee827d573dea5649e4dd1b69f95951aebe40684aabe12757fc583add6 2024-02-21T23:46:16.584 INFO:journalctl@ceph.mds.e.smithi176.stdout:Feb 21 23:46:16 smithi176 systemd[1]: Started Ceph mds.e for 7ed81218-d112-11ee-95bd-87774f69a715. 2024-02-21T23:46:16.889 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:16 smithi178 ceph-mon[47547]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:16.889 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:16 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/1910151909' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-21T23:46:16.889 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:16 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:16.889 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:16 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:16.889 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:16 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:16.890 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:16 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-21T23:46:16.890 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:16 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-21T23:46:16.890 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:16 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:16.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:16 smithi090 ceph-mon[43385]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:16.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:16 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/1910151909' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-21T23:46:16.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:16 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:16.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:16 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:16.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:16 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:16.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:16 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-21T23:46:16.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:16 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-21T23:46:16.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:16 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:17.083 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:16 smithi176 ceph-mon[47172]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:17.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:16 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/1910151909' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-21T23:46:17.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:16 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:17.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:16 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:17.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:16 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:17.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:16 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-21T23:46:17.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:16 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-21T23:46:17.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:16 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:17.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:17 smithi178 ceph-mon[47547]: Deploying daemon mds.f on smithi178 2024-02-21T23:46:17.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:17 smithi178 ceph-mon[47547]: mds.? [v2:172.21.15.176:6834/3430010314,v1:172.21.15.176:6835/3430010314] up:boot 2024-02-21T23:46:17.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:17 smithi178 ceph-mon[47547]: fsmap 5 up:standby 2024-02-21T23:46:17.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:17 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-02-21T23:46:17.925 INFO:journalctl@ceph.mds.f.smithi178.stdout:Feb 21 23:46:17 smithi178 systemd[1]: Starting Ceph mds.f for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:46:17.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:17 smithi090 ceph-mon[43385]: Deploying daemon mds.f on smithi178 2024-02-21T23:46:17.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:17 smithi090 ceph-mon[43385]: mds.? [v2:172.21.15.176:6834/3430010314,v1:172.21.15.176:6835/3430010314] up:boot 2024-02-21T23:46:17.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:17 smithi090 ceph-mon[43385]: fsmap 5 up:standby 2024-02-21T23:46:17.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:17 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-02-21T23:46:18.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:17 smithi176 ceph-mon[47172]: Deploying daemon mds.f on smithi178 2024-02-21T23:46:18.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:17 smithi176 ceph-mon[47172]: mds.? [v2:172.21.15.176:6834/3430010314,v1:172.21.15.176:6835/3430010314] up:boot 2024-02-21T23:46:18.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:17 smithi176 ceph-mon[47172]: fsmap 5 up:standby 2024-02-21T23:46:18.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:17 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-02-21T23:46:18.111 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:46:18.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.261+0000 7fa890937700 1 -- 172.21.15.90:0/2952816386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa88c100ce0 msgr2=0x7fa88c101140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:18.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.261+0000 7fa890937700 1 --2- 172.21.15.90:0/2952816386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa88c100ce0 0x7fa88c101140 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa8740099c0 tx=0x7fa874009cd0 comp rx=0 tx=0).stop 2024-02-21T23:46:18.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.261+0000 7fa890937700 1 -- 172.21.15.90:0/2952816386 shutdown_connections 2024-02-21T23:46:18.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.261+0000 7fa890937700 1 --2- 172.21.15.90:0/2952816386 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa88c101680 0x7fa88c107f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:18.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.261+0000 7fa890937700 1 --2- 172.21.15.90:0/2952816386 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa88c0ffaf0 0x7fa88c0fff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:18.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.261+0000 7fa890937700 1 --2- 172.21.15.90:0/2952816386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa88c100ce0 0x7fa88c101140 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:18.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.261+0000 7fa890937700 1 -- 172.21.15.90:0/2952816386 >> 172.21.15.90:0/2952816386 conn(0x7fa88c0fb060 msgr2=0x7fa88c0fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:18.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.261+0000 7fa890937700 1 -- 172.21.15.90:0/2952816386 shutdown_connections 2024-02-21T23:46:18.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.261+0000 7fa890937700 1 -- 172.21.15.90:0/2952816386 wait complete. 2024-02-21T23:46:18.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.262+0000 7fa890937700 1 Processor -- start 2024-02-21T23:46:18.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.262+0000 7fa890937700 1 -- start start 2024-02-21T23:46:18.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.262+0000 7fa890937700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa88c0ffaf0 0x7fa88c0688a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:18.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.262+0000 7fa890937700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa88c100ce0 0x7fa88c19b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:18.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.262+0000 7fa890937700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa88c101680 0x7fa88c19de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:18.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.262+0000 7fa890937700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa88c19e630 con 0x7fa88c101680 2024-02-21T23:46:18.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.262+0000 7fa890937700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa88c19e7a0 con 0x7fa88c100ce0 2024-02-21T23:46:18.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.262+0000 7fa890937700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa88c19e960 con 0x7fa88c0ffaf0 2024-02-21T23:46:18.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.262+0000 7fa88ad9d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa88c101680 0x7fa88c19de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:18.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.262+0000 7fa88a59c700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa88c0ffaf0 0x7fa88c0688a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:18.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.262+0000 7fa88ad9d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa88c101680 0x7fa88c19de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52596/0 (socket says 172.21.15.90:52596) 2024-02-21T23:46:18.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.262+0000 7fa88ad9d700 1 -- 172.21.15.90:0/413759270 learned_addr learned my addr 172.21.15.90:0/413759270 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:18.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.262+0000 7fa889d9b700 1 --2- 172.21.15.90:0/413759270 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa88c100ce0 0x7fa88c19b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:18.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.262+0000 7fa88ad9d700 1 -- 172.21.15.90:0/413759270 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa88c0ffaf0 msgr2=0x7fa88c0688a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:18.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.262+0000 7fa88ad9d700 1 --2- 172.21.15.90:0/413759270 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa88c0ffaf0 0x7fa88c0688a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:18.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.262+0000 7fa88ad9d700 1 -- 172.21.15.90:0/413759270 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa88c100ce0 msgr2=0x7fa88c19b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:18.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.262+0000 7fa88ad9d700 1 --2- 172.21.15.90:0/413759270 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa88c100ce0 0x7fa88c19b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:18.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.262+0000 7fa88ad9d700 1 -- 172.21.15.90:0/413759270 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8740095f0 con 0x7fa88c101680 2024-02-21T23:46:18.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.262+0000 7fa889d9b700 1 --2- 172.21.15.90:0/413759270 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa88c100ce0 0x7fa88c19b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:46:18.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.263+0000 7fa88ad9d700 1 --2- 172.21.15.90:0/413759270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa88c101680 0x7fa88c19de90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa884007ee0 tx=0x7fa88400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:18.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.263+0000 7fa8837fe700 1 -- 172.21.15.90:0/413759270 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa884017070 con 0x7fa88c101680 2024-02-21T23:46:18.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.263+0000 7fa8837fe700 1 -- 172.21.15.90:0/413759270 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa88400f040 con 0x7fa88c101680 2024-02-21T23:46:18.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.263+0000 7fa890937700 1 -- 172.21.15.90:0/413759270 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa88c19ec40 con 0x7fa88c101680 2024-02-21T23:46:18.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.263+0000 7fa8837fe700 1 -- 172.21.15.90:0/413759270 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa884005740 con 0x7fa88c101680 2024-02-21T23:46:18.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.263+0000 7fa890937700 1 -- 172.21.15.90:0/413759270 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa88c19f100 con 0x7fa88c101680 2024-02-21T23:46:18.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.264+0000 7fa890937700 1 -- 172.21.15.90:0/413759270 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa88c106f10 con 0x7fa88c101680 2024-02-21T23:46:18.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.265+0000 7fa8837fe700 1 -- 172.21.15.90:0/413759270 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7fa88400dc60 con 0x7fa88c101680 2024-02-21T23:46:18.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.265+0000 7fa8837fe700 1 --2- 172.21.15.90:0/413759270 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fa8780a1120 0x7fa8780a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:18.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.265+0000 7fa8837fe700 1 -- 172.21.15.90:0/413759270 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa8840cbc00 con 0x7fa88c101680 2024-02-21T23:46:18.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.265+0000 7fa88a59c700 1 --2- 172.21.15.90:0/413759270 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fa8780a1120 0x7fa8780a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:18.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.266+0000 7fa88a59c700 1 --2- 172.21.15.90:0/413759270 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fa8780a1120 0x7fa8780a35e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa88c100b40 tx=0x7fa87c007400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:18.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.270+0000 7fa8837fe700 1 -- 172.21.15.90:0/413759270 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fa8840d2080 con 0x7fa88c101680 2024-02-21T23:46:18.424 INFO:journalctl@ceph.mds.f.smithi178.stdout:Feb 21 23:46:18 smithi178 ceph-mds[85047]: -- 172.21.15.178:0/2650211509 <== mon.1 v2:172.21.15.178:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558eea9691e0 con 0x558eea93f000 2024-02-21T23:46:18.424 INFO:journalctl@ceph.mds.f.smithi178.stdout:Feb 21 23:46:18 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-f[85024]: starting mds.f at 2024-02-21T23:46:18.424 INFO:journalctl@ceph.mds.f.smithi178.stdout:Feb 21 23:46:18 smithi178 bash[85009]: e50e160d8df63db12d82cc20497ccaa7f0afba1d756c0c515486e0b40ecccfde 2024-02-21T23:46:18.424 INFO:journalctl@ceph.mds.f.smithi178.stdout:Feb 21 23:46:18 smithi178 systemd[1]: Started Ceph mds.f for 7ed81218-d112-11ee-95bd-87774f69a715. 2024-02-21T23:46:18.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.523+0000 7fa890937700 1 -- 172.21.15.90:0/413759270 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa88c02cc70 con 0x7fa88c101680 2024-02-21T23:46:18.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.524+0000 7fa8837fe700 1 -- 172.21.15.90:0/413759270 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4294 (secure 0 0 0) 0x7fa884095830 con 0x7fa88c101680 2024-02-21T23:46:18.508 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:46:18.509 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/1591404300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1591404300},{"type":"v1","addr":"172.21.15.90:6835","nonce":1591404300}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/1441666263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":1441666263},{"type":"v1","addr":"172.21.15.90:6837","nonce":1441666263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1403694104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1403694104},{"type":"v1","addr":"172.21.15.178:6833","nonce":1403694104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/2956287083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":2956287083},{"type":"v1","addr":"172.21.15.176:6833","nonce":2956287083}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/3430010314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":3430010314},{"type":"v1","addr":"172.21.15.176:6835","nonce":3430010314}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[]} 2024-02-21T23:46:18.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.526+0000 7fa890937700 1 -- 172.21.15.90:0/413759270 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fa8780a1120 msgr2=0x7fa8780a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:18.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.526+0000 7fa890937700 1 --2- 172.21.15.90:0/413759270 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fa8780a1120 0x7fa8780a35e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa88c100b40 tx=0x7fa87c007400 comp rx=0 tx=0).stop 2024-02-21T23:46:18.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.526+0000 7fa890937700 1 -- 172.21.15.90:0/413759270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa88c101680 msgr2=0x7fa88c19de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:18.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.526+0000 7fa890937700 1 --2- 172.21.15.90:0/413759270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa88c101680 0x7fa88c19de90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa884007ee0 tx=0x7fa88400d3b0 comp rx=0 tx=0).stop 2024-02-21T23:46:18.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.526+0000 7fa890937700 1 -- 172.21.15.90:0/413759270 shutdown_connections 2024-02-21T23:46:18.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.526+0000 7fa890937700 1 --2- 172.21.15.90:0/413759270 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fa8780a1120 0x7fa8780a35e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:18.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.526+0000 7fa890937700 1 --2- 172.21.15.90:0/413759270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa88c101680 0x7fa88c19de90 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:18.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.526+0000 7fa890937700 1 --2- 172.21.15.90:0/413759270 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa88c0ffaf0 0x7fa88c0688a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:18.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.526+0000 7fa890937700 1 --2- 172.21.15.90:0/413759270 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa88c100ce0 0x7fa88c19b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:18.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.526+0000 7fa890937700 1 -- 172.21.15.90:0/413759270 >> 172.21.15.90:0/413759270 conn(0x7fa88c0fb060 msgr2=0x7fa88c104050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:18.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.526+0000 7fa890937700 1 -- 172.21.15.90:0/413759270 shutdown_connections 2024-02-21T23:46:18.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:18.526+0000 7fa890937700 1 -- 172.21.15.90:0/413759270 wait complete. 2024-02-21T23:46:18.511 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 6 2024-02-21T23:46:18.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:18 smithi178 ceph-mon[47547]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:18.874 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:18 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:18.874 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:18 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:18.874 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:18 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:18.874 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:18 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-21T23:46:18.874 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:18 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-21T23:46:18.874 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:18 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:18.874 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:18 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/413759270' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-21T23:46:18.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:18 smithi090 ceph-mon[43385]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:18.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:18 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:18.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:18 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:18.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:18 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:18.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:18 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-21T23:46:18.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:18 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-21T23:46:18.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:18 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:18.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:18 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/413759270' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-21T23:46:19.083 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:18 smithi176 ceph-mon[47172]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:19.083 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:18 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:19.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:18 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:19.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:18 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:19.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:18 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-21T23:46:19.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:18 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-21T23:46:19.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:18 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:19.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:18 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/413759270' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-21T23:46:19.562 INFO:journalctl@ceph.mds.c.smithi178.stdout:Feb 21 23:46:19 smithi178 systemd[1]: Starting Ceph mds.c for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:46:19.835 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:19 smithi178 ceph-mon[47547]: Deploying daemon mds.c on smithi178 2024-02-21T23:46:19.836 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:19 smithi178 ceph-mon[47547]: mds.? [v2:172.21.15.178:6834/1260308729,v1:172.21.15.178:6835/1260308729] up:boot 2024-02-21T23:46:19.836 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:19 smithi178 ceph-mon[47547]: fsmap 6 up:standby 2024-02-21T23:46:19.836 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:19 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-02-21T23:46:19.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:19 smithi090 ceph-mon[43385]: Deploying daemon mds.c on smithi178 2024-02-21T23:46:19.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:19 smithi090 ceph-mon[43385]: mds.? [v2:172.21.15.178:6834/1260308729,v1:172.21.15.178:6835/1260308729] up:boot 2024-02-21T23:46:19.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:19 smithi090 ceph-mon[43385]: fsmap 6 up:standby 2024-02-21T23:46:19.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:19 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-02-21T23:46:20.083 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:19 smithi176 ceph-mon[47172]: Deploying daemon mds.c on smithi178 2024-02-21T23:46:20.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:19 smithi176 ceph-mon[47172]: mds.? [v2:172.21.15.178:6834/1260308729,v1:172.21.15.178:6835/1260308729] up:boot 2024-02-21T23:46:20.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:19 smithi176 ceph-mon[47172]: fsmap 6 up:standby 2024-02-21T23:46:20.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:19 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-02-21T23:46:20.423 INFO:journalctl@ceph.mds.c.smithi178.stdout:Feb 21 23:46:20 smithi178 ceph-mds[85390]: -- 172.21.15.178:0/1023596159 <== mon.1 v2:172.21.15.178:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555b9b6191e0 con 0x555b9b5ee800 2024-02-21T23:46:20.424 INFO:journalctl@ceph.mds.c.smithi178.stdout:Feb 21 23:46:20 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-c[85367]: starting mds.c at 2024-02-21T23:46:20.424 INFO:journalctl@ceph.mds.c.smithi178.stdout:Feb 21 23:46:20 smithi178 bash[85351]: 638fd5cf304a53f1e276613988a7329ba24e38240e44ff130df8438c45041384 2024-02-21T23:46:20.424 INFO:journalctl@ceph.mds.c.smithi178.stdout:Feb 21 23:46:20 smithi178 systemd[1]: Started Ceph mds.c for 7ed81218-d112-11ee-95bd-87774f69a715. 2024-02-21T23:46:20.524 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:46:20.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.676+0000 7f962f841700 1 -- 172.21.15.90:0/3495214948 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f96280ff490 msgr2=0x7f96281018c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:20.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.676+0000 7f962f841700 1 --2- 172.21.15.90:0/3495214948 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f96280ff490 0x7f96281018c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9624009990 tx=0x7f9624009ca0 comp rx=0 tx=0).stop 2024-02-21T23:46:20.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.676+0000 7f962f841700 1 -- 172.21.15.90:0/3495214948 shutdown_connections 2024-02-21T23:46:20.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.676+0000 7f962f841700 1 --2- 172.21.15.90:0/3495214948 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96281047d0 0x7f9628104c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:20.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.676+0000 7f962f841700 1 --2- 172.21.15.90:0/3495214948 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f96280ff490 0x7f96281018c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:20.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.676+0000 7f962f841700 1 --2- 172.21.15.90:0/3495214948 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9628101e00 0x7f9628104290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:20.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.676+0000 7f962f841700 1 -- 172.21.15.90:0/3495214948 >> 172.21.15.90:0/3495214948 conn(0x7f96280fb090 msgr2=0x7f96280fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:20.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.676+0000 7f962f841700 1 -- 172.21.15.90:0/3495214948 shutdown_connections 2024-02-21T23:46:20.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.676+0000 7f962f841700 1 -- 172.21.15.90:0/3495214948 wait complete. 2024-02-21T23:46:20.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.676+0000 7f962f841700 1 Processor -- start 2024-02-21T23:46:20.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.676+0000 7f962f841700 1 -- start start 2024-02-21T23:46:20.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.677+0000 7f962f841700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f96280ff490 0x7f962819d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:20.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.677+0000 7f962f841700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9628101e00 0x7f962819d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:20.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.677+0000 7f962f841700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f96281047d0 0x7f96281a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:20.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.677+0000 7f962f841700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96281a4c90 con 0x7f96281047d0 2024-02-21T23:46:20.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.677+0000 7f962f841700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96281a4dd0 con 0x7f9628101e00 2024-02-21T23:46:20.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.677+0000 7f962f841700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96281a4f60 con 0x7f96280ff490 2024-02-21T23:46:20.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.677+0000 7f962ddde700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f96281047d0 0x7f96281a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:20.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.677+0000 7f962cddc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9628101e00 0x7f962819d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:20.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.677+0000 7f962ddde700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f96281047d0 0x7f96281a46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52616/0 (socket says 172.21.15.90:52616) 2024-02-21T23:46:20.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.677+0000 7f962cddc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9628101e00 0x7f962819d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.90:49422/0 (socket says 172.21.15.90:49422) 2024-02-21T23:46:20.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.677+0000 7f962ddde700 1 -- 172.21.15.90:0/2890430957 learned_addr learned my addr 172.21.15.90:0/2890430957 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:20.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.677+0000 7f962d5dd700 1 --2- 172.21.15.90:0/2890430957 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f96280ff490 0x7f962819d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:20.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.677+0000 7f962ddde700 1 -- 172.21.15.90:0/2890430957 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f96280ff490 msgr2=0x7f962819d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:20.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.677+0000 7f962ddde700 1 --2- 172.21.15.90:0/2890430957 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f96280ff490 0x7f962819d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:20.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.677+0000 7f962ddde700 1 -- 172.21.15.90:0/2890430957 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9628101e00 msgr2=0x7f962819d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:20.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.677+0000 7f962ddde700 1 --2- 172.21.15.90:0/2890430957 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9628101e00 0x7f962819d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:20.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.677+0000 7f962ddde700 1 -- 172.21.15.90:0/2890430957 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96240095f0 con 0x7f96281047d0 2024-02-21T23:46:20.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.677+0000 7f962cddc700 1 --2- 172.21.15.90:0/2890430957 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9628101e00 0x7f962819d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:46:20.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.677+0000 7f962ddde700 1 --2- 172.21.15.90:0/2890430957 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f96281047d0 0x7f96281a46c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f962000ef30 tx=0x7f962000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:20.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.677+0000 7f961e7fc700 1 -- 172.21.15.90:0/2890430957 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9620018070 con 0x7f96281047d0 2024-02-21T23:46:20.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.677+0000 7f961e7fc700 1 -- 172.21.15.90:0/2890430957 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f962000f040 con 0x7f96281047d0 2024-02-21T23:46:20.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.677+0000 7f962f841700 1 -- 172.21.15.90:0/2890430957 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96281a51e0 con 0x7f96281047d0 2024-02-21T23:46:20.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.677+0000 7f961e7fc700 1 -- 172.21.15.90:0/2890430957 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962000cd00 con 0x7f96281047d0 2024-02-21T23:46:20.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.677+0000 7f962f841700 1 -- 172.21.15.90:0/2890430957 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96281a56f0 con 0x7f96281047d0 2024-02-21T23:46:20.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.678+0000 7f962f841700 1 -- 172.21.15.90:0/2890430957 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96280fcc10 con 0x7f96281047d0 2024-02-21T23:46:20.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.679+0000 7f961e7fc700 1 -- 172.21.15.90:0/2890430957 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7f9620003a00 con 0x7f96281047d0 2024-02-21T23:46:20.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.680+0000 7f961e7fc700 1 --2- 172.21.15.90:0/2890430957 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f96140a11a0 0x7f96140a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:20.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.680+0000 7f961e7fc700 1 -- 172.21.15.90:0/2890430957 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f96200cbf10 con 0x7f96281047d0 2024-02-21T23:46:20.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.680+0000 7f962d5dd700 1 --2- 172.21.15.90:0/2890430957 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f96140a11a0 0x7f96140a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:20.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.680+0000 7f962d5dd700 1 --2- 172.21.15.90:0/2890430957 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f96140a11a0 0x7f96140a3660 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f962400b2e0 tx=0x7f9624005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:20.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.684+0000 7f961e7fc700 1 -- 172.21.15.90:0/2890430957 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f9620095bd0 con 0x7f96281047d0 2024-02-21T23:46:20.785 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:20 smithi176 ceph-mon[47172]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:20.786 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:20 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:20.786 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:20 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:20.786 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:20 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:20.786 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:20 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-21T23:46:20.786 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:20 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-21T23:46:20.786 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:20 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:20.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.929+0000 7f962f841700 1 -- 172.21.15.90:0/2890430957 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9628067110 con 0x7f96281047d0 2024-02-21T23:46:20.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.930+0000 7f961e7fc700 1 -- 172.21.15.90:0/2890430957 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+5814 (secure 0 0 0) 0x7f96200991f0 con 0x7f96281047d0 2024-02-21T23:46:20.915 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:46:20.915 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/1591404300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1591404300},{"type":"v1","addr":"172.21.15.90:6835","nonce":1591404300}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/1441666263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":1441666263},{"type":"v1","addr":"172.21.15.90:6837","nonce":1441666263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1403694104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1403694104},{"type":"v1","addr":"172.21.15.178:6833","nonce":1403694104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/2956287083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":2956287083},{"type":"v1","addr":"172.21.15.176:6833","nonce":2956287083}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/3430010314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":3430010314},{"type":"v1","addr":"172.21.15.176:6835","nonce":3430010314}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/1260308729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1260308729},{"type":"v1","addr":"172.21.15.178:6835","nonce":1260308729}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/4166803229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":4166803229},{"type":"v1","addr":"172.21.15.178:6837","nonce":4166803229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2024-02-21T23:46:20.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.932+0000 7f962f841700 1 -- 172.21.15.90:0/2890430957 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f96140a11a0 msgr2=0x7f96140a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:20.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.932+0000 7f962f841700 1 --2- 172.21.15.90:0/2890430957 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f96140a11a0 0x7f96140a3660 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f962400b2e0 tx=0x7f9624005fd0 comp rx=0 tx=0).stop 2024-02-21T23:46:20.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.932+0000 7f962f841700 1 -- 172.21.15.90:0/2890430957 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f96281047d0 msgr2=0x7f96281a46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:20.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.932+0000 7f962f841700 1 --2- 172.21.15.90:0/2890430957 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f96281047d0 0x7f96281a46c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f962000ef30 tx=0x7f962000c3a0 comp rx=0 tx=0).stop 2024-02-21T23:46:20.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.932+0000 7f962f841700 1 -- 172.21.15.90:0/2890430957 shutdown_connections 2024-02-21T23:46:20.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.932+0000 7f962f841700 1 --2- 172.21.15.90:0/2890430957 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f96140a11a0 0x7f96140a3660 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:20.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.932+0000 7f962f841700 1 --2- 172.21.15.90:0/2890430957 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f96281047d0 0x7f96281a46c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:20.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.932+0000 7f962f841700 1 --2- 172.21.15.90:0/2890430957 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f96280ff490 0x7f962819d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:20.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.932+0000 7f962f841700 1 --2- 172.21.15.90:0/2890430957 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9628101e00 0x7f962819d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:20.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.932+0000 7f962f841700 1 -- 172.21.15.90:0/2890430957 >> 172.21.15.90:0/2890430957 conn(0x7f96280fb090 msgr2=0x7f9628102880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:20.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.932+0000 7f962f841700 1 -- 172.21.15.90:0/2890430957 shutdown_connections 2024-02-21T23:46:20.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:20.932+0000 7f962f841700 1 -- 172.21.15.90:0/2890430957 wait complete. 2024-02-21T23:46:20.918 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 8 2024-02-21T23:46:20.923 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:20 smithi178 ceph-mon[47547]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:20.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:20 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:20.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:20 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:20.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:20 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:20.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:20 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-21T23:46:20.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:20 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-21T23:46:20.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:20 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:20.927 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:20 smithi090 ceph-mon[43385]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:20.927 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:20 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:20.927 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:20 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:20.928 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:20 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:20.928 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:20 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-21T23:46:20.928 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:20 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-21T23:46:20.928 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:20 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:21.334 INFO:journalctl@ceph.mds.h.smithi176.stdout:Feb 21 23:46:21 smithi176 systemd[1]: Starting Ceph mds.h for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:46:21.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:21 smithi178 ceph-mon[47547]: Deploying daemon mds.h on smithi176 2024-02-21T23:46:21.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:21 smithi178 ceph-mon[47547]: mds.? [v2:172.21.15.178:6836/4166803229,v1:172.21.15.178:6837/4166803229] up:boot 2024-02-21T23:46:21.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:21 smithi178 ceph-mon[47547]: fsmap 7 up:standby 2024-02-21T23:46:21.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:21 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-02-21T23:46:21.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:21 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/2890430957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-21T23:46:21.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:21 smithi090 ceph-mon[43385]: Deploying daemon mds.h on smithi176 2024-02-21T23:46:21.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:21 smithi090 ceph-mon[43385]: mds.? [v2:172.21.15.178:6836/4166803229,v1:172.21.15.178:6837/4166803229] up:boot 2024-02-21T23:46:21.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:21 smithi090 ceph-mon[43385]: fsmap 7 up:standby 2024-02-21T23:46:21.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:21 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-02-21T23:46:21.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:21 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/2890430957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-21T23:46:22.083 INFO:journalctl@ceph.mds.h.smithi176.stdout:Feb 21 23:46:21 smithi176 ceph-mds[83301]: -- 172.21.15.176:0/3310812036 <== mon.2 v2:172.21.15.176:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c5ac6411e0 con 0x55c5ac617000 2024-02-21T23:46:22.083 INFO:journalctl@ceph.mds.h.smithi176.stdout:Feb 21 23:46:21 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-h[83278]: starting mds.h at 2024-02-21T23:46:22.083 INFO:journalctl@ceph.mds.h.smithi176.stdout:Feb 21 23:46:21 smithi176 bash[83264]: 1385499c12c35ffec69fec1ec679e08b40780785942df755b964b1504d4da25b 2024-02-21T23:46:22.084 INFO:journalctl@ceph.mds.h.smithi176.stdout:Feb 21 23:46:21 smithi176 systemd[1]: Started Ceph mds.h for 7ed81218-d112-11ee-95bd-87774f69a715. 2024-02-21T23:46:22.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:21 smithi176 ceph-mon[47172]: Deploying daemon mds.h on smithi176 2024-02-21T23:46:22.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:21 smithi176 ceph-mon[47172]: mds.? [v2:172.21.15.178:6836/4166803229,v1:172.21.15.178:6837/4166803229] up:boot 2024-02-21T23:46:22.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:21 smithi176 ceph-mon[47172]: fsmap 7 up:standby 2024-02-21T23:46:22.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:21 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-02-21T23:46:22.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:21 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/2890430957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-21T23:46:22.842 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:22 smithi178 ceph-mon[47547]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:22.842 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:22 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:22.842 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:22 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:22.842 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:22 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:22.842 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:22 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-21T23:46:22.843 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:22 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-21T23:46:22.843 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:22 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:22.930 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:46:22.959 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:22 smithi090 ceph-mon[43385]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:22.960 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:22 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:22.960 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:22 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:22.960 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:22 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:22.960 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:22 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-21T23:46:22.960 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:22 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-21T23:46:22.960 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:22 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:23.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.082+0000 7f2536617700 1 -- 172.21.15.90:0/2843611756 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2530101e20 msgr2=0x7f2530102280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:23.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.082+0000 7f2536617700 1 --2- 172.21.15.90:0/2843611756 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2530101e20 0x7f2530102280 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f25180099c0 tx=0x7f2518009cd0 comp rx=0 tx=0).stop 2024-02-21T23:46:23.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.082+0000 7f2536617700 1 -- 172.21.15.90:0/2843611756 shutdown_connections 2024-02-21T23:46:23.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.082+0000 7f2536617700 1 --2- 172.21.15.90:0/2843611756 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f25301027c0 0x7f2530109070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:23.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.082+0000 7f2536617700 1 --2- 172.21.15.90:0/2843611756 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2530100c30 0x7f2530101050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:23.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.082+0000 7f2536617700 1 --2- 172.21.15.90:0/2843611756 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2530101e20 0x7f2530102280 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:23.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.082+0000 7f2536617700 1 -- 172.21.15.90:0/2843611756 >> 172.21.15.90:0/2843611756 conn(0x7f25300fc1a0 msgr2=0x7f25300fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:23.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.082+0000 7f2536617700 1 -- 172.21.15.90:0/2843611756 shutdown_connections 2024-02-21T23:46:23.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.082+0000 7f2536617700 1 -- 172.21.15.90:0/2843611756 wait complete. 2024-02-21T23:46:23.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.082+0000 7f2536617700 1 Processor -- start 2024-02-21T23:46:23.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.083+0000 7f2536617700 1 -- start start 2024-02-21T23:46:23.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.083+0000 7f2536617700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2530100c30 0x7f2530074ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:23.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.083+0000 7f2536617700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2530101e20 0x7f2530073160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:23.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.083+0000 7f2536617700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25301027c0 0x7f25300736a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:23.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.083+0000 7f2536617700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2530073e40 con 0x7f25301027c0 2024-02-21T23:46:23.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.083+0000 7f2536617700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2530073fb0 con 0x7f2530100c30 2024-02-21T23:46:23.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.083+0000 7f2536617700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25301a0b60 con 0x7f2530101e20 2024-02-21T23:46:23.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.083+0000 7f2534bb4700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25301027c0 0x7f25300736a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:23.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.083+0000 7f2534bb4700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25301027c0 0x7f25300736a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52628/0 (socket says 172.21.15.90:52628) 2024-02-21T23:46:23.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.083+0000 7f2534bb4700 1 -- 172.21.15.90:0/1146565118 learned_addr learned my addr 172.21.15.90:0/1146565118 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:23.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.083+0000 7f2527fff700 1 --2- 172.21.15.90:0/1146565118 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2530101e20 0x7f2530073160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:23.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.083+0000 7f252ffff700 1 --2- 172.21.15.90:0/1146565118 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2530100c30 0x7f2530074ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:23.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.083+0000 7f2534bb4700 1 -- 172.21.15.90:0/1146565118 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2530101e20 msgr2=0x7f2530073160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:23.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.083+0000 7f2534bb4700 1 --2- 172.21.15.90:0/1146565118 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2530101e20 0x7f2530073160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:23.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.083+0000 7f2534bb4700 1 -- 172.21.15.90:0/1146565118 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2530100c30 msgr2=0x7f2530074ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:23.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.083+0000 7f2534bb4700 1 --2- 172.21.15.90:0/1146565118 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2530100c30 0x7f2530074ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:23.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.083+0000 7f2534bb4700 1 -- 172.21.15.90:0/1146565118 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25180095f0 con 0x7f25301027c0 2024-02-21T23:46:23.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.083+0000 7f2534bb4700 1 --2- 172.21.15.90:0/1146565118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25301027c0 0x7f25300736a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2528007ee0 tx=0x7f252800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:23.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.083+0000 7f252dffb700 1 -- 172.21.15.90:0/1146565118 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2528017070 con 0x7f25301027c0 2024-02-21T23:46:23.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.083+0000 7f2536617700 1 -- 172.21.15.90:0/1146565118 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25301a0df0 con 0x7f25301027c0 2024-02-21T23:46:23.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.083+0000 7f252dffb700 1 -- 172.21.15.90:0/1146565118 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f252800f040 con 0x7f25301027c0 2024-02-21T23:46:23.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.083+0000 7f252dffb700 1 -- 172.21.15.90:0/1146565118 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25280054b0 con 0x7f25301027c0 2024-02-21T23:46:23.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.083+0000 7f2536617700 1 -- 172.21.15.90:0/1146565118 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25301a1220 con 0x7f25301027c0 2024-02-21T23:46:23.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.084+0000 7f2536617700 1 -- 172.21.15.90:0/1146565118 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2530108060 con 0x7f25301027c0 2024-02-21T23:46:23.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.085+0000 7f252dffb700 1 -- 172.21.15.90:0/1146565118 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7f2528004ad0 con 0x7f25301027c0 2024-02-21T23:46:23.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.086+0000 7f252dffb700 1 --2- 172.21.15.90:0/1146565118 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f251c0a1120 0x7f251c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:23.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.086+0000 7f252ffff700 1 --2- 172.21.15.90:0/1146565118 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f251c0a1120 0x7f251c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:23.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.086+0000 7f252dffb700 1 -- 172.21.15.90:0/1146565118 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f25280cc1f0 con 0x7f25301027c0 2024-02-21T23:46:23.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.086+0000 7f252ffff700 1 --2- 172.21.15.90:0/1146565118 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f251c0a1120 0x7f251c0a35e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2530101c80 tx=0x7f2520009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:23.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.090+0000 7f252dffb700 1 -- 172.21.15.90:0/1146565118 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f2528095e20 con 0x7f25301027c0 2024-02-21T23:46:23.083 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:22 smithi176 ceph-mon[47172]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:23.083 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:22 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:23.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:22 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:23.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:22 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:23.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:22 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-21T23:46:23.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:22 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-21T23:46:23.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:22 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:23.095 INFO:journalctl@ceph.mds.i.smithi178.stdout:Feb 21 23:46:23 smithi178 systemd[1]: Starting Ceph mds.i for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:46:23.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.332+0000 7f2536617700 1 -- 172.21.15.90:0/1146565118 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2530067dc0 con 0x7f25301027c0 2024-02-21T23:46:23.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.332+0000 7f252dffb700 1 -- 172.21.15.90:0/1146565118 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6574 (secure 0 0 0) 0x7f2528099440 con 0x7f25301027c0 2024-02-21T23:46:23.317 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:46:23.317 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/1591404300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1591404300},{"type":"v1","addr":"172.21.15.90:6835","nonce":1591404300}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/1441666263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":1441666263},{"type":"v1","addr":"172.21.15.90:6837","nonce":1441666263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1403694104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1403694104},{"type":"v1","addr":"172.21.15.178:6833","nonce":1403694104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/2956287083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":2956287083},{"type":"v1","addr":"172.21.15.176:6833","nonce":2956287083}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/3430010314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":3430010314},{"type":"v1","addr":"172.21.15.176:6835","nonce":3430010314}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/1260308729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1260308729},{"type":"v1","addr":"172.21.15.178:6835","nonce":1260308729}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/4166803229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":4166803229},{"type":"v1","addr":"172.21.15.178:6837","nonce":4166803229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24458,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/4127802836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":4127802836},{"type":"v1","addr":"172.21.15.176:6837","nonce":4127802836}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2024-02-21T23:46:23.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.334+0000 7f2536617700 1 -- 172.21.15.90:0/1146565118 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f251c0a1120 msgr2=0x7f251c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:23.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.334+0000 7f2536617700 1 --2- 172.21.15.90:0/1146565118 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f251c0a1120 0x7f251c0a35e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2530101c80 tx=0x7f2520009260 comp rx=0 tx=0).stop 2024-02-21T23:46:23.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.334+0000 7f2536617700 1 -- 172.21.15.90:0/1146565118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25301027c0 msgr2=0x7f25300736a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:23.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.334+0000 7f2536617700 1 --2- 172.21.15.90:0/1146565118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25301027c0 0x7f25300736a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2528007ee0 tx=0x7f252800d3b0 comp rx=0 tx=0).stop 2024-02-21T23:46:23.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.334+0000 7f2536617700 1 -- 172.21.15.90:0/1146565118 shutdown_connections 2024-02-21T23:46:23.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.334+0000 7f2536617700 1 --2- 172.21.15.90:0/1146565118 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f251c0a1120 0x7f251c0a35e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:23.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.334+0000 7f2536617700 1 --2- 172.21.15.90:0/1146565118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25301027c0 0x7f25300736a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:23.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.334+0000 7f2536617700 1 --2- 172.21.15.90:0/1146565118 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2530100c30 0x7f2530074ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:23.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.334+0000 7f2536617700 1 --2- 172.21.15.90:0/1146565118 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2530101e20 0x7f2530073160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:23.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.334+0000 7f2536617700 1 -- 172.21.15.90:0/1146565118 >> 172.21.15.90:0/1146565118 conn(0x7f25300fc1a0 msgr2=0x7f25301051a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:23.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.335+0000 7f2536617700 1 -- 172.21.15.90:0/1146565118 shutdown_connections 2024-02-21T23:46:23.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:23.335+0000 7f2536617700 1 -- 172.21.15.90:0/1146565118 wait complete. 2024-02-21T23:46:23.320 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 9 2024-02-21T23:46:23.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:23 smithi178 ceph-mon[47547]: Deploying daemon mds.i on smithi178 2024-02-21T23:46:23.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:23 smithi178 ceph-mon[47547]: mds.? [v2:172.21.15.176:6836/4127802836,v1:172.21.15.176:6837/4127802836] up:boot 2024-02-21T23:46:23.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:23 smithi178 ceph-mon[47547]: fsmap 8 up:standby 2024-02-21T23:46:23.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:23 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-02-21T23:46:23.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:23 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/1146565118' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-21T23:46:23.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:23 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:23.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:23 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:23.924 INFO:journalctl@ceph.mds.i.smithi178.stdout:Feb 21 23:46:23 smithi178 ceph-mds[85729]: -- 172.21.15.178:0/26619399 <== mon.1 v2:172.21.15.178:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562939e0b1e0 con 0x562939de0c00 2024-02-21T23:46:23.925 INFO:journalctl@ceph.mds.i.smithi178.stdout:Feb 21 23:46:23 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-i[85706]: starting mds.i at 2024-02-21T23:46:23.925 INFO:journalctl@ceph.mds.i.smithi178.stdout:Feb 21 23:46:23 smithi178 bash[85692]: d7d409a740ce52aad30707bfb53989e855a92c0ec6b7d171e8e78c3f705a0821 2024-02-21T23:46:23.925 INFO:journalctl@ceph.mds.i.smithi178.stdout:Feb 21 23:46:23 smithi178 systemd[1]: Started Ceph mds.i for 7ed81218-d112-11ee-95bd-87774f69a715. 2024-02-21T23:46:23.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:23 smithi090 ceph-mon[43385]: Deploying daemon mds.i on smithi178 2024-02-21T23:46:23.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:23 smithi090 ceph-mon[43385]: mds.? [v2:172.21.15.176:6836/4127802836,v1:172.21.15.176:6837/4127802836] up:boot 2024-02-21T23:46:23.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:23 smithi090 ceph-mon[43385]: fsmap 8 up:standby 2024-02-21T23:46:23.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:23 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-02-21T23:46:23.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:23 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/1146565118' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-21T23:46:23.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:23 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:23.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:23 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:24.083 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:23 smithi176 ceph-mon[47172]: Deploying daemon mds.i on smithi178 2024-02-21T23:46:24.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:23 smithi176 ceph-mon[47172]: mds.? [v2:172.21.15.176:6836/4127802836,v1:172.21.15.176:6837/4127802836] up:boot 2024-02-21T23:46:24.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:23 smithi176 ceph-mon[47172]: fsmap 8 up:standby 2024-02-21T23:46:24.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:23 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-02-21T23:46:24.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:23 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/1146565118' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-21T23:46:24.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:23 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:24.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:23 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:24.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:24 smithi178 ceph-mon[47547]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:24.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:24 smithi178 ceph-mon[47547]: mds.? [v2:172.21.15.178:6838/3726236229,v1:172.21.15.178:6839/3726236229] up:boot 2024-02-21T23:46:24.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:24 smithi178 ceph-mon[47547]: fsmap 9 up:standby 2024-02-21T23:46:24.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:24 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-02-21T23:46:24.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:24 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:24.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:24 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-21T23:46:24.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:24 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-21T23:46:24.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:24 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:24.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:24 smithi178 ceph-mon[47547]: Deploying daemon mds.g on smithi090 2024-02-21T23:46:24.953 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:24 smithi090 ceph-mon[43385]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:24.953 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:24 smithi090 ceph-mon[43385]: mds.? [v2:172.21.15.178:6838/3726236229,v1:172.21.15.178:6839/3726236229] up:boot 2024-02-21T23:46:24.953 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:24 smithi090 ceph-mon[43385]: fsmap 9 up:standby 2024-02-21T23:46:24.953 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:24 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-02-21T23:46:24.953 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:24 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:24.953 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:24 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-21T23:46:24.953 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:24 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-21T23:46:24.954 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:24 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:24.954 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:24 smithi090 ceph-mon[43385]: Deploying daemon mds.g on smithi090 2024-02-21T23:46:25.083 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:24 smithi176 ceph-mon[47172]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:25.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:24 smithi176 ceph-mon[47172]: mds.? [v2:172.21.15.178:6838/3726236229,v1:172.21.15.178:6839/3726236229] up:boot 2024-02-21T23:46:25.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:24 smithi176 ceph-mon[47172]: fsmap 9 up:standby 2024-02-21T23:46:25.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:24 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-02-21T23:46:25.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:24 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:25.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:24 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-21T23:46:25.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:24 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-21T23:46:25.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:24 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:25.084 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:24 smithi176 ceph-mon[47172]: Deploying daemon mds.g on smithi090 2024-02-21T23:46:25.332 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:46:25.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.482+0000 7f20b5116700 1 -- 172.21.15.90:0/4059780710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20b0074db0 msgr2=0x7f20b0073200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:25.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.482+0000 7f20b5116700 1 --2- 172.21.15.90:0/4059780710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20b0074db0 0x7f20b0073200 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f20a0009990 tx=0x7f20a0009ca0 comp rx=0 tx=0).stop 2024-02-21T23:46:25.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.482+0000 7f20b5116700 1 -- 172.21.15.90:0/4059780710 shutdown_connections 2024-02-21T23:46:25.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.482+0000 7f20b5116700 1 --2- 172.21.15.90:0/4059780710 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20b0108af0 0x7f20b010af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:25.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.482+0000 7f20b5116700 1 --2- 172.21.15.90:0/4059780710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20b0074db0 0x7f20b0073200 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:25.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.482+0000 7f20b5116700 1 --2- 172.21.15.90:0/4059780710 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f20b00737d0 0x7f20b0073c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:25.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.482+0000 7f20b5116700 1 -- 172.21.15.90:0/4059780710 >> 172.21.15.90:0/4059780710 conn(0x7f20b00fc1a0 msgr2=0x7f20b00fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:25.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.482+0000 7f20b5116700 1 -- 172.21.15.90:0/4059780710 shutdown_connections 2024-02-21T23:46:25.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.482+0000 7f20b5116700 1 -- 172.21.15.90:0/4059780710 wait complete. 2024-02-21T23:46:25.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.483+0000 7f20b5116700 1 Processor -- start 2024-02-21T23:46:25.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.483+0000 7f20b5116700 1 -- start start 2024-02-21T23:46:25.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.483+0000 7f20b5116700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f20b00737d0 0x7f20b0198d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:25.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.483+0000 7f20b5116700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20b0074db0 0x7f20b01992a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:25.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.483+0000 7f20b5116700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20b0108af0 0x7f20b01a0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:25.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.483+0000 7f20b5116700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b01a0850 con 0x7f20b0108af0 2024-02-21T23:46:25.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.483+0000 7f20b5116700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b01a09c0 con 0x7f20b0074db0 2024-02-21T23:46:25.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.483+0000 7f20b5116700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b01a0b80 con 0x7f20b00737d0 2024-02-21T23:46:25.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.483+0000 7f20af59e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20b0108af0 0x7f20b01a0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:25.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.483+0000 7f20af59e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20b0108af0 0x7f20b01a0280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52656/0 (socket says 172.21.15.90:52656) 2024-02-21T23:46:25.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.483+0000 7f20af59e700 1 -- 172.21.15.90:0/2217780842 learned_addr learned my addr 172.21.15.90:0/2217780842 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:25.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.483+0000 7f20aed9d700 1 --2- 172.21.15.90:0/2217780842 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f20b00737d0 0x7f20b0198d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:25.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.483+0000 7f20ae59c700 1 --2- 172.21.15.90:0/2217780842 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20b0074db0 0x7f20b01992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:25.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.484+0000 7f20af59e700 1 -- 172.21.15.90:0/2217780842 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f20b00737d0 msgr2=0x7f20b0198d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:25.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.484+0000 7f20af59e700 1 --2- 172.21.15.90:0/2217780842 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f20b00737d0 0x7f20b0198d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:25.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.484+0000 7f20af59e700 1 -- 172.21.15.90:0/2217780842 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20b0074db0 msgr2=0x7f20b01992a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:25.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.484+0000 7f20af59e700 1 --2- 172.21.15.90:0/2217780842 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20b0074db0 0x7f20b01992a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:25.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.484+0000 7f20af59e700 1 -- 172.21.15.90:0/2217780842 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a00095f0 con 0x7f20b0108af0 2024-02-21T23:46:25.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.484+0000 7f20ae59c700 1 --2- 172.21.15.90:0/2217780842 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20b0074db0 0x7f20b01992a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:46:25.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.484+0000 7f20af59e700 1 --2- 172.21.15.90:0/2217780842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20b0108af0 0x7f20b01a0280 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f20a8007ee0 tx=0x7f20a800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:25.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.484+0000 7f2097fff700 1 -- 172.21.15.90:0/2217780842 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a8017070 con 0x7f20b0108af0 2024-02-21T23:46:25.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.484+0000 7f2097fff700 1 -- 172.21.15.90:0/2217780842 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f20a800f040 con 0x7f20b0108af0 2024-02-21T23:46:25.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.484+0000 7f20b5116700 1 -- 172.21.15.90:0/2217780842 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20b01a0e60 con 0x7f20b0108af0 2024-02-21T23:46:25.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.484+0000 7f2097fff700 1 -- 172.21.15.90:0/2217780842 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a8005740 con 0x7f20b0108af0 2024-02-21T23:46:25.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.484+0000 7f20b5116700 1 -- 172.21.15.90:0/2217780842 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20b01a1280 con 0x7f20b0108af0 2024-02-21T23:46:25.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.485+0000 7f20b5116700 1 -- 172.21.15.90:0/2217780842 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20b01078b0 con 0x7f20b0108af0 2024-02-21T23:46:25.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.486+0000 7f2097fff700 1 -- 172.21.15.90:0/2217780842 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7f20a800dc60 con 0x7f20b0108af0 2024-02-21T23:46:25.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.487+0000 7f2097fff700 1 --2- 172.21.15.90:0/2217780842 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f20980a11a0 0x7f20980a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:25.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.487+0000 7f20aed9d700 1 --2- 172.21.15.90:0/2217780842 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f20980a11a0 0x7f20980a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:25.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.487+0000 7f2097fff700 1 -- 172.21.15.90:0/2217780842 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f20a80cc330 con 0x7f20b0108af0 2024-02-21T23:46:25.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.487+0000 7f20aed9d700 1 --2- 172.21.15.90:0/2217780842 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f20980a11a0 0x7f20980a3660 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f20a000b680 tx=0x7f20a0005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:25.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.490+0000 7f2097fff700 1 -- 172.21.15.90:0/2217780842 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f20a8095ec0 con 0x7f20b0108af0 2024-02-21T23:46:25.727 INFO:journalctl@ceph.mds.g.smithi090.stdout:Feb 21 23:46:25 smithi090 bash[94764]: b9c6cc518d9b55fe777c3291d14ed6bd87cf7103a3c777ff36b1c3d20b3ce36d 2024-02-21T23:46:25.727 INFO:journalctl@ceph.mds.g.smithi090.stdout:Feb 21 23:46:25 smithi090 systemd[1]: Started Ceph mds.g for 7ed81218-d112-11ee-95bd-87774f69a715. 2024-02-21T23:46:25.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.751+0000 7f20b5116700 1 -- 172.21.15.90:0/2217780842 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20b0067df0 con 0x7f20b0108af0 2024-02-21T23:46:25.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.752+0000 7f2097fff700 1 -- 172.21.15.90:0/2217780842 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8091 (secure 0 0 0) 0x7f20a80b9030 con 0x7f20b0108af0 2024-02-21T23:46:25.736 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:46:25.736 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/1591404300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1591404300},{"type":"v1","addr":"172.21.15.90:6835","nonce":1591404300}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/1441666263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":1441666263},{"type":"v1","addr":"172.21.15.90:6837","nonce":1441666263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14607,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6839/732535159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":732535159},{"type":"v1","addr":"172.21.15.90:6839","nonce":732535159}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1403694104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1403694104},{"type":"v1","addr":"172.21.15.178:6833","nonce":1403694104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/2956287083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":2956287083},{"type":"v1","addr":"172.21.15.176:6833","nonce":2956287083}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/3430010314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":3430010314},{"type":"v1","addr":"172.21.15.176:6835","nonce":3430010314}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/1260308729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1260308729},{"type":"v1","addr":"172.21.15.178:6835","nonce":1260308729}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/4166803229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":4166803229},{"type":"v1","addr":"172.21.15.178:6837","nonce":4166803229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24458,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/4127802836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":4127802836},{"type":"v1","addr":"172.21.15.176:6837","nonce":4127802836}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24470,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6839/3726236229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":3726236229},{"type":"v1","addr":"172.21.15.178:6839","nonce":3726236229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2024-02-21T23:46:25.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.754+0000 7f20b5116700 1 -- 172.21.15.90:0/2217780842 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f20980a11a0 msgr2=0x7f20980a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:25.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.754+0000 7f20b5116700 1 --2- 172.21.15.90:0/2217780842 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f20980a11a0 0x7f20980a3660 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f20a000b680 tx=0x7f20a0005fd0 comp rx=0 tx=0).stop 2024-02-21T23:46:25.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.754+0000 7f20b5116700 1 -- 172.21.15.90:0/2217780842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20b0108af0 msgr2=0x7f20b01a0280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:25.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.754+0000 7f20b5116700 1 --2- 172.21.15.90:0/2217780842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20b0108af0 0x7f20b01a0280 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f20a8007ee0 tx=0x7f20a800d3b0 comp rx=0 tx=0).stop 2024-02-21T23:46:25.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.754+0000 7f20b5116700 1 -- 172.21.15.90:0/2217780842 shutdown_connections 2024-02-21T23:46:25.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.754+0000 7f20b5116700 1 --2- 172.21.15.90:0/2217780842 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f20980a11a0 0x7f20980a3660 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:25.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.754+0000 7f20b5116700 1 --2- 172.21.15.90:0/2217780842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20b0108af0 0x7f20b01a0280 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:25.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.754+0000 7f20b5116700 1 --2- 172.21.15.90:0/2217780842 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f20b00737d0 0x7f20b0198d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:25.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.754+0000 7f20b5116700 1 --2- 172.21.15.90:0/2217780842 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20b0074db0 0x7f20b01992a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:25.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.754+0000 7f20b5116700 1 -- 172.21.15.90:0/2217780842 >> 172.21.15.90:0/2217780842 conn(0x7f20b00fc1a0 msgr2=0x7f20b01049f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:25.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.754+0000 7f20b5116700 1 -- 172.21.15.90:0/2217780842 shutdown_connections 2024-02-21T23:46:25.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:25.754+0000 7f20b5116700 1 -- 172.21.15.90:0/2217780842 wait complete. 2024-02-21T23:46:25.740 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 11 2024-02-21T23:46:26.713 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:26 smithi090 ceph-mon[43385]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:26.713 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:26 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:26.713 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:26 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:26.713 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:26 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:26.713 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:26 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-21T23:46:26.713 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:26 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-21T23:46:26.713 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:26 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:26.714 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:26 smithi090 ceph-mon[43385]: Deploying daemon mds.a on smithi090 2024-02-21T23:46:26.714 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:26 smithi090 ceph-mon[43385]: mds.? [v2:172.21.15.90:6838/732535159,v1:172.21.15.90:6839/732535159] up:boot 2024-02-21T23:46:26.714 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:26 smithi090 ceph-mon[43385]: fsmap 10 up:standby 2024-02-21T23:46:26.714 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:26 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-02-21T23:46:26.714 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:26 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/2217780842' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-21T23:46:26.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:26 smithi176 ceph-mon[47172]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:26.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:26 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:26.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:26 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:26.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:26 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:26.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:26 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-21T23:46:26.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:26 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-21T23:46:26.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:26 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:26.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:26 smithi176 ceph-mon[47172]: Deploying daemon mds.a on smithi090 2024-02-21T23:46:26.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:26 smithi176 ceph-mon[47172]: mds.? [v2:172.21.15.90:6838/732535159,v1:172.21.15.90:6839/732535159] up:boot 2024-02-21T23:46:26.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:26 smithi176 ceph-mon[47172]: fsmap 10 up:standby 2024-02-21T23:46:26.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:26 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-02-21T23:46:26.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:26 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/2217780842' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-21T23:46:26.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:26 smithi178 ceph-mon[47547]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:26.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:26 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:26.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:26 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:26.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:26 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:26.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:26 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-21T23:46:26.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:26 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-21T23:46:26.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:26 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:26.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:26 smithi178 ceph-mon[47547]: Deploying daemon mds.a on smithi090 2024-02-21T23:46:26.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:26 smithi178 ceph-mon[47547]: mds.? [v2:172.21.15.90:6838/732535159,v1:172.21.15.90:6839/732535159] up:boot 2024-02-21T23:46:26.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:26 smithi178 ceph-mon[47547]: fsmap 10 up:standby 2024-02-21T23:46:26.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:26 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-02-21T23:46:26.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:26 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/2217780842' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-21T23:46:26.977 INFO:journalctl@ceph.mds.a.smithi090.stdout:Feb 21 23:46:26 smithi090 systemd[1]: Starting Ceph mds.a for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:46:27.727 INFO:journalctl@ceph.mds.a.smithi090.stdout:Feb 21 23:46:27 smithi090 bash[95153]: 249b72f98c9a755605bc852d74e71a69cb81fabb56b943f7a1b1536b7cb27897 2024-02-21T23:46:27.727 INFO:journalctl@ceph.mds.a.smithi090.stdout:Feb 21 23:46:27 smithi090 systemd[1]: Started Ceph mds.a for 7ed81218-d112-11ee-95bd-87774f69a715. 2024-02-21T23:46:27.727 INFO:journalctl@ceph.mds.a.smithi090.stdout:Feb 21 23:46:27 smithi090 ceph-mds[95191]: -- 172.21.15.90:0/3636172502 <== mon.0 v2:172.21.15.90:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f8163471e0 con 0x55f81631cc00 2024-02-21T23:46:27.727 INFO:journalctl@ceph.mds.a.smithi090.stdout:Feb 21 23:46:27 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-a[95168]: starting mds.a at 2024-02-21T23:46:27.753 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:46:27.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.897+0000 7f27849c1700 1 -- 172.21.15.90:0/2745247718 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2780106900 msgr2=0x7f2780108d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:27.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.897+0000 7f27849c1700 1 --2- 172.21.15.90:0/2745247718 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2780106900 0x7f2780108d70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f277800b0a0 tx=0x7f277800b3b0 comp rx=0 tx=0).stop 2024-02-21T23:46:27.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.898+0000 7f27849c1700 1 -- 172.21.15.90:0/2745247718 shutdown_connections 2024-02-21T23:46:27.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.898+0000 7f27849c1700 1 --2- 172.21.15.90:0/2745247718 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2780106900 0x7f2780108d70 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:27.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.898+0000 7f27849c1700 1 --2- 172.21.15.90:0/2745247718 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f27801016a0 0x7f2780103a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:27.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.898+0000 7f27849c1700 1 --2- 172.21.15.90:0/2745247718 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2780103fd0 0x7f27801063c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:27.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.898+0000 7f27849c1700 1 -- 172.21.15.90:0/2745247718 >> 172.21.15.90:0/2745247718 conn(0x7f27800fb090 msgr2=0x7f27800fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:27.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.898+0000 7f27849c1700 1 -- 172.21.15.90:0/2745247718 shutdown_connections 2024-02-21T23:46:27.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.898+0000 7f27849c1700 1 -- 172.21.15.90:0/2745247718 wait complete. 2024-02-21T23:46:27.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.898+0000 7f27849c1700 1 Processor -- start 2024-02-21T23:46:27.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.898+0000 7f27849c1700 1 -- start start 2024-02-21T23:46:27.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.898+0000 7f27849c1700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27801016a0 0x7f2780196c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:27.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.899+0000 7f27849c1700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2780103fd0 0x7f2780197150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:27.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.899+0000 7f27849c1700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2780106900 0x7f278019e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:27.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.899+0000 7f27849c1700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f278019e740 con 0x7f27801016a0 2024-02-21T23:46:27.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.899+0000 7f27849c1700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f278019e880 con 0x7f2780103fd0 2024-02-21T23:46:27.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.899+0000 7f27849c1700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f278019ea10 con 0x7f2780106900 2024-02-21T23:46:27.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.899+0000 7f277dd9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2780103fd0 0x7f2780197150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:27.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.899+0000 7f277dd9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2780103fd0 0x7f2780197150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.90:49506/0 (socket says 172.21.15.90:49506) 2024-02-21T23:46:27.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.899+0000 7f277dd9b700 1 -- 172.21.15.90:0/853207830 learned_addr learned my addr 172.21.15.90:0/853207830 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:27.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.899+0000 7f277e59c700 1 --2- 172.21.15.90:0/853207830 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27801016a0 0x7f2780196c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:27.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.899+0000 7f277ed9d700 1 --2- 172.21.15.90:0/853207830 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2780106900 0x7f278019e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:27.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.899+0000 7f277dd9b700 1 -- 172.21.15.90:0/853207830 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2780106900 msgr2=0x7f278019e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:27.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.899+0000 7f277dd9b700 1 --2- 172.21.15.90:0/853207830 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2780106900 0x7f278019e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:27.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.899+0000 7f277dd9b700 1 -- 172.21.15.90:0/853207830 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27801016a0 msgr2=0x7f2780196c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:27.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.899+0000 7f277dd9b700 1 --2- 172.21.15.90:0/853207830 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27801016a0 0x7f2780196c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:27.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.899+0000 7f277dd9b700 1 -- 172.21.15.90:0/853207830 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2778009d00 con 0x7f2780103fd0 2024-02-21T23:46:27.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.899+0000 7f277dd9b700 1 --2- 172.21.15.90:0/853207830 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2780103fd0 0x7f2780197150 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f277000e9a0 tx=0x7f277000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:27.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.899+0000 7f276f7fe700 1 -- 172.21.15.90:0/853207830 <== mon.2 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2770009790 con 0x7f2780103fd0 2024-02-21T23:46:27.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.899+0000 7f27849c1700 1 -- 172.21.15.90:0/853207830 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f278019ecf0 con 0x7f2780103fd0 2024-02-21T23:46:27.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.899+0000 7f276f7fe700 1 -- 172.21.15.90:0/853207830 <== mon.2 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f277000cbd0 con 0x7f2780103fd0 2024-02-21T23:46:27.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.899+0000 7f276f7fe700 1 -- 172.21.15.90:0/853207830 <== mon.2 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2770018680 con 0x7f2780103fd0 2024-02-21T23:46:27.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.899+0000 7f27849c1700 1 -- 172.21.15.90:0/853207830 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f278019f1a0 con 0x7f2780103fd0 2024-02-21T23:46:27.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.900+0000 7f27849c1700 1 -- 172.21.15.90:0/853207830 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27800fcc10 con 0x7f2780103fd0 2024-02-21T23:46:27.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.901+0000 7f276f7fe700 1 -- 172.21.15.90:0/853207830 <== mon.2 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7f277000cd40 con 0x7f2780103fd0 2024-02-21T23:46:27.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.902+0000 7f276f7fe700 1 --2- 172.21.15.90:0/853207830 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f27680a1120 0x7f27680a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:27.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.902+0000 7f276f7fe700 1 -- 172.21.15.90:0/853207830 <== mon.2 v2:172.21.15.176:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f2770014070 con 0x7f2780103fd0 2024-02-21T23:46:27.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.902+0000 7f277e59c700 1 --2- 172.21.15.90:0/853207830 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f27680a1120 0x7f27680a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:27.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.902+0000 7f277e59c700 1 --2- 172.21.15.90:0/853207830 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f27680a1120 0x7f27680a35e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2774007c60 tx=0x7f27740073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:27.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:27.906+0000 7f276f7fe700 1 -- 172.21.15.90:0/853207830 <== mon.2 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f2770096320 con 0x7f2780103fd0 2024-02-21T23:46:28.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:28.145+0000 7f27849c1700 1 -- 172.21.15.90:0/853207830 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2780067110 con 0x7f2780103fd0 2024-02-21T23:46:28.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:28.146+0000 7f276f7fe700 1 -- 172.21.15.90:0/853207830 <== mon.2 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+8849 (secure 0 0 0) 0x7f2770099940 con 0x7f2780103fd0 2024-02-21T23:46:28.131 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:46:28.131 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/1591404300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1591404300},{"type":"v1","addr":"172.21.15.90:6835","nonce":1591404300}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/1441666263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":1441666263},{"type":"v1","addr":"172.21.15.90:6837","nonce":1441666263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14607,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6839/732535159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":732535159},{"type":"v1","addr":"172.21.15.90:6839","nonce":732535159}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14619,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/3572584154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":3572584154},{"type":"v1","addr":"172.21.15.90:6841","nonce":3572584154}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1403694104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1403694104},{"type":"v1","addr":"172.21.15.178:6833","nonce":1403694104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/2956287083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":2956287083},{"type":"v1","addr":"172.21.15.176:6833","nonce":2956287083}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/3430010314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":3430010314},{"type":"v1","addr":"172.21.15.176:6835","nonce":3430010314}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/1260308729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1260308729},{"type":"v1","addr":"172.21.15.178:6835","nonce":1260308729}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/4166803229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":4166803229},{"type":"v1","addr":"172.21.15.178:6837","nonce":4166803229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24458,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/4127802836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":4127802836},{"type":"v1","addr":"172.21.15.176:6837","nonce":4127802836}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24470,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6839/3726236229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":3726236229},{"type":"v1","addr":"172.21.15.178:6839","nonce":3726236229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2024-02-21T23:46:28.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:28.148+0000 7f27849c1700 1 -- 172.21.15.90:0/853207830 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f27680a1120 msgr2=0x7f27680a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:28.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:28.148+0000 7f27849c1700 1 --2- 172.21.15.90:0/853207830 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f27680a1120 0x7f27680a35e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2774007c60 tx=0x7f27740073d0 comp rx=0 tx=0).stop 2024-02-21T23:46:28.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:28.148+0000 7f27849c1700 1 -- 172.21.15.90:0/853207830 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2780103fd0 msgr2=0x7f2780197150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:28.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:28.148+0000 7f27849c1700 1 --2- 172.21.15.90:0/853207830 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2780103fd0 0x7f2780197150 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f277000e9a0 tx=0x7f277000ecb0 comp rx=0 tx=0).stop 2024-02-21T23:46:28.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:28.148+0000 7f27849c1700 1 -- 172.21.15.90:0/853207830 shutdown_connections 2024-02-21T23:46:28.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:28.148+0000 7f27849c1700 1 --2- 172.21.15.90:0/853207830 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f27680a1120 0x7f27680a35e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:28.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:28.148+0000 7f27849c1700 1 --2- 172.21.15.90:0/853207830 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2780106900 0x7f278019e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:28.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:28.148+0000 7f27849c1700 1 --2- 172.21.15.90:0/853207830 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27801016a0 0x7f2780196c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:28.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:28.148+0000 7f27849c1700 1 --2- 172.21.15.90:0/853207830 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2780103fd0 0x7f2780197150 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:28.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:28.148+0000 7f27849c1700 1 -- 172.21.15.90:0/853207830 >> 172.21.15.90:0/853207830 conn(0x7f27800fb090 msgr2=0x7f2780102000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:28.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:28.148+0000 7f27849c1700 1 -- 172.21.15.90:0/853207830 shutdown_connections 2024-02-21T23:46:28.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:28.148+0000 7f27849c1700 1 -- 172.21.15.90:0/853207830 wait complete. 2024-02-21T23:46:28.134 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 12 2024-02-21T23:46:28.355 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:28 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:28.355 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:28 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:28.355 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:28 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:28.355 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:28 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-21T23:46:28.355 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:28 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-21T23:46:28.355 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:28 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:28.355 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:28 smithi176 ceph-mon[47172]: Deploying daemon mds.b on smithi176 2024-02-21T23:46:28.355 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:28 smithi176 ceph-mon[47172]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:28.355 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:28 smithi176 ceph-mon[47172]: mds.? [v2:172.21.15.90:6840/3572584154,v1:172.21.15.90:6841/3572584154] up:boot 2024-02-21T23:46:28.355 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:28 smithi176 ceph-mon[47172]: fsmap 11 up:standby 2024-02-21T23:46:28.355 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:28 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-02-21T23:46:28.356 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:28 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/853207830' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-21T23:46:28.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:28 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:28.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:28 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:28.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:28 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:28.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:28 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-21T23:46:28.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:28 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-21T23:46:28.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:28 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:28.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:28 smithi090 ceph-mon[43385]: Deploying daemon mds.b on smithi176 2024-02-21T23:46:28.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:28 smithi090 ceph-mon[43385]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:28.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:28 smithi090 ceph-mon[43385]: mds.? [v2:172.21.15.90:6840/3572584154,v1:172.21.15.90:6841/3572584154] up:boot 2024-02-21T23:46:28.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:28 smithi090 ceph-mon[43385]: fsmap 11 up:standby 2024-02-21T23:46:28.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:28 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-02-21T23:46:28.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:28 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/853207830' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-21T23:46:28.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:28 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:28.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:28 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:28.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:28 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:28.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:28 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-21T23:46:28.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:28 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-21T23:46:28.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:28 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:28.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:28 smithi178 ceph-mon[47547]: Deploying daemon mds.b on smithi176 2024-02-21T23:46:28.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:28 smithi178 ceph-mon[47547]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:28.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:28 smithi178 ceph-mon[47547]: mds.? [v2:172.21.15.90:6840/3572584154,v1:172.21.15.90:6841/3572584154] up:boot 2024-02-21T23:46:28.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:28 smithi178 ceph-mon[47547]: fsmap 11 up:standby 2024-02-21T23:46:28.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:28 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-02-21T23:46:28.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:28 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/853207830' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-21T23:46:28.996 INFO:journalctl@ceph.mds.b.smithi176.stdout:Feb 21 23:46:28 smithi176 systemd[1]: Starting Ceph mds.b for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:46:29.334 INFO:journalctl@ceph.mds.b.smithi176.stdout:Feb 21 23:46:29 smithi176 ceph-mds[83640]: -- 172.21.15.176:0/3614580843 <== mon.2 v2:172.21.15.176:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b4e1a531e0 con 0x55b4e1a28800 2024-02-21T23:46:29.334 INFO:journalctl@ceph.mds.b.smithi176.stdout:Feb 21 23:46:29 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-b[83617]: starting mds.b at 2024-02-21T23:46:29.334 INFO:journalctl@ceph.mds.b.smithi176.stdout:Feb 21 23:46:29 smithi176 bash[83602]: 7206681e35cd75d537603065004c7592314f8f10e35079cc34dc8a0e1e2d07fe 2024-02-21T23:46:29.334 INFO:journalctl@ceph.mds.b.smithi176.stdout:Feb 21 23:46:29 smithi176 systemd[1]: Started Ceph mds.b for 7ed81218-d112-11ee-95bd-87774f69a715. 2024-02-21T23:46:30.146 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:46:30.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.310+0000 7fdfa4adf700 1 -- 172.21.15.90:0/574982015 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfa0101e80 msgr2=0x7fdfa01022e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:30.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.310+0000 7fdfa4adf700 1 --2- 172.21.15.90:0/574982015 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfa0101e80 0x7fdfa01022e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fdf94009990 tx=0x7fdf94009ca0 comp rx=0 tx=0).stop 2024-02-21T23:46:30.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.310+0000 7fdfa4adf700 1 -- 172.21.15.90:0/574982015 shutdown_connections 2024-02-21T23:46:30.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.310+0000 7fdfa4adf700 1 --2- 172.21.15.90:0/574982015 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdfa0102820 0x7fdfa01090d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:30.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.310+0000 7fdfa4adf700 1 --2- 172.21.15.90:0/574982015 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdfa00feeb0 0x7fdfa00ff2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:30.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.310+0000 7fdfa4adf700 1 --2- 172.21.15.90:0/574982015 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfa0101e80 0x7fdfa01022e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:30.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.310+0000 7fdfa4adf700 1 -- 172.21.15.90:0/574982015 >> 172.21.15.90:0/574982015 conn(0x7fdfa00fa7e0 msgr2=0x7fdfa00fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:30.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.310+0000 7fdfa4adf700 1 -- 172.21.15.90:0/574982015 shutdown_connections 2024-02-21T23:46:30.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.310+0000 7fdfa4adf700 1 -- 172.21.15.90:0/574982015 wait complete. 2024-02-21T23:46:30.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.310+0000 7fdfa4adf700 1 Processor -- start 2024-02-21T23:46:30.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.310+0000 7fdfa4adf700 1 -- start start 2024-02-21T23:46:30.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.311+0000 7fdfa4adf700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdfa00feeb0 0x7fdfa0196b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:30.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.311+0000 7fdfa4adf700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdfa0101e80 0x7fdfa0197090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:30.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.311+0000 7fdfa4adf700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfa0102820 0x7fdfa019e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:30.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.311+0000 7fdfa4adf700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa019e680 con 0x7fdfa0102820 2024-02-21T23:46:30.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.311+0000 7fdfa4adf700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa019e7c0 con 0x7fdfa00feeb0 2024-02-21T23:46:30.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.311+0000 7fdfa4adf700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa019e950 con 0x7fdfa0101e80 2024-02-21T23:46:30.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.311+0000 7fdf9ed9d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfa0102820 0x7fdfa019e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:30.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.311+0000 7fdf9ed9d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfa0102820 0x7fdfa019e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41530/0 (socket says 172.21.15.90:41530) 2024-02-21T23:46:30.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.311+0000 7fdf9ed9d700 1 -- 172.21.15.90:0/1647674777 learned_addr learned my addr 172.21.15.90:0/1647674777 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:30.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.311+0000 7fdf9e59c700 1 --2- 172.21.15.90:0/1647674777 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdfa00feeb0 0x7fdfa0196b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:30.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.311+0000 7fdf9dd9b700 1 --2- 172.21.15.90:0/1647674777 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdfa0101e80 0x7fdfa0197090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:30.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.311+0000 7fdf9ed9d700 1 -- 172.21.15.90:0/1647674777 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdfa0101e80 msgr2=0x7fdfa0197090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:30.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.311+0000 7fdf9ed9d700 1 --2- 172.21.15.90:0/1647674777 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdfa0101e80 0x7fdfa0197090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:30.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.311+0000 7fdf9ed9d700 1 -- 172.21.15.90:0/1647674777 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdfa00feeb0 msgr2=0x7fdfa0196b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:30.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.311+0000 7fdf9ed9d700 1 --2- 172.21.15.90:0/1647674777 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdfa00feeb0 0x7fdfa0196b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:30.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.311+0000 7fdf9ed9d700 1 -- 172.21.15.90:0/1647674777 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf940095f0 con 0x7fdfa0102820 2024-02-21T23:46:30.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.311+0000 7fdf9ed9d700 1 --2- 172.21.15.90:0/1647674777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfa0102820 0x7fdfa019e0b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdf9800ef30 tx=0x7fdf9800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:30.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.311+0000 7fdf8f7fe700 1 -- 172.21.15.90:0/1647674777 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf98018070 con 0x7fdfa0102820 2024-02-21T23:46:30.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.311+0000 7fdf8f7fe700 1 -- 172.21.15.90:0/1647674777 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdf9800f040 con 0x7fdfa0102820 2024-02-21T23:46:30.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.311+0000 7fdfa4adf700 1 -- 172.21.15.90:0/1647674777 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfa019ebd0 con 0x7fdfa0102820 2024-02-21T23:46:30.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.311+0000 7fdf8f7fe700 1 -- 172.21.15.90:0/1647674777 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf9800cd00 con 0x7fdfa0102820 2024-02-21T23:46:30.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.312+0000 7fdfa4adf700 1 -- 172.21.15.90:0/1647674777 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfa019f130 con 0x7fdfa0102820 2024-02-21T23:46:30.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.312+0000 7fdfa4adf700 1 -- 172.21.15.90:0/1647674777 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfa0108160 con 0x7fdfa0102820 2024-02-21T23:46:30.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.314+0000 7fdf8f7fe700 1 -- 172.21.15.90:0/1647674777 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7fdf98003a00 con 0x7fdfa0102820 2024-02-21T23:46:30.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.315+0000 7fdf8f7fe700 1 --2- 172.21.15.90:0/1647674777 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fdf880a11a0 0x7fdf880a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:30.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.315+0000 7fdf9e59c700 1 --2- 172.21.15.90:0/1647674777 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fdf880a11a0 0x7fdf880a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:30.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.315+0000 7fdf8f7fe700 1 -- 172.21.15.90:0/1647674777 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fdf980cc860 con 0x7fdfa0102820 2024-02-21T23:46:30.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.315+0000 7fdf9e59c700 1 --2- 172.21.15.90:0/1647674777 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fdf880a11a0 0x7fdf880a3660 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdfa0067b80 tx=0x7fdf90009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:30.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.318+0000 7fdf8f7fe700 1 -- 172.21.15.90:0/1647674777 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fdf98096520 con 0x7fdfa0102820 2024-02-21T23:46:30.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:30 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:30.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:30 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:30.335 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:30 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:30.335 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:30 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:30.335 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:30 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:30.335 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:30 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:46:30.335 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:30 smithi176 ceph-mon[47172]: mds.? [v2:172.21.15.176:6838/875627757,v1:172.21.15.176:6839/875627757] up:boot 2024-02-21T23:46:30.335 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:30 smithi176 ceph-mon[47172]: fsmap 12 up:standby 2024-02-21T23:46:30.335 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:30 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-02-21T23:46:30.335 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:30 smithi176 ceph-mon[47172]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:30.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:30 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:30.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:30 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:30.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:30 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:30.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:30 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:30.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:30 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:30.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:30 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:46:30.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:30 smithi178 ceph-mon[47547]: mds.? [v2:172.21.15.176:6838/875627757,v1:172.21.15.176:6839/875627757] up:boot 2024-02-21T23:46:30.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:30 smithi178 ceph-mon[47547]: fsmap 12 up:standby 2024-02-21T23:46:30.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:30 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-02-21T23:46:30.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:30 smithi178 ceph-mon[47547]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:30.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:30 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:30.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:30 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:30.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:30 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:30.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:30 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:30.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:30 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:30.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:30 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:46:30.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:30 smithi090 ceph-mon[43385]: mds.? [v2:172.21.15.176:6838/875627757,v1:172.21.15.176:6839/875627757] up:boot 2024-02-21T23:46:30.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:30 smithi090 ceph-mon[43385]: fsmap 12 up:standby 2024-02-21T23:46:30.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:30 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-02-21T23:46:30.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:30 smithi090 ceph-mon[43385]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:30.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.581+0000 7fdfa4adf700 1 -- 172.21.15.90:0/1647674777 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdfa0067110 con 0x7fdfa0102820 2024-02-21T23:46:30.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.582+0000 7fdf8f7fe700 1 -- 172.21.15.90:0/1647674777 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+9607 (secure 0 0 0) 0x7fdf98025380 con 0x7fdfa0102820 2024-02-21T23:46:30.566 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:46:30.567 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/1591404300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1591404300},{"type":"v1","addr":"172.21.15.90:6835","nonce":1591404300}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/1441666263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":1441666263},{"type":"v1","addr":"172.21.15.90:6837","nonce":1441666263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14607,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6839/732535159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":732535159},{"type":"v1","addr":"172.21.15.90:6839","nonce":732535159}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14619,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/3572584154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":3572584154},{"type":"v1","addr":"172.21.15.90:6841","nonce":3572584154}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1403694104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1403694104},{"type":"v1","addr":"172.21.15.178:6833","nonce":1403694104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/2956287083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":2956287083},{"type":"v1","addr":"172.21.15.176:6833","nonce":2956287083}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/3430010314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":3430010314},{"type":"v1","addr":"172.21.15.176:6835","nonce":3430010314}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/1260308729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1260308729},{"type":"v1","addr":"172.21.15.178:6835","nonce":1260308729}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/4166803229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":4166803229},{"type":"v1","addr":"172.21.15.178:6837","nonce":4166803229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24458,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/4127802836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":4127802836},{"type":"v1","addr":"172.21.15.176:6837","nonce":4127802836}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24470,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6839/3726236229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":3726236229},{"type":"v1","addr":"172.21.15.178:6839","nonce":3726236229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24500,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6839/875627757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6838","nonce":875627757},{"type":"v1","addr":"172.21.15.176:6839","nonce":875627757}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[]} 2024-02-21T23:46:30.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.584+0000 7fdfa4adf700 1 -- 172.21.15.90:0/1647674777 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fdf880a11a0 msgr2=0x7fdf880a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:30.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.584+0000 7fdfa4adf700 1 --2- 172.21.15.90:0/1647674777 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fdf880a11a0 0x7fdf880a3660 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdfa0067b80 tx=0x7fdf90009260 comp rx=0 tx=0).stop 2024-02-21T23:46:30.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.584+0000 7fdfa4adf700 1 -- 172.21.15.90:0/1647674777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfa0102820 msgr2=0x7fdfa019e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:30.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.584+0000 7fdfa4adf700 1 --2- 172.21.15.90:0/1647674777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfa0102820 0x7fdfa019e0b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdf9800ef30 tx=0x7fdf9800c3a0 comp rx=0 tx=0).stop 2024-02-21T23:46:30.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.584+0000 7fdfa4adf700 1 -- 172.21.15.90:0/1647674777 shutdown_connections 2024-02-21T23:46:30.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.584+0000 7fdfa4adf700 1 --2- 172.21.15.90:0/1647674777 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fdf880a11a0 0x7fdf880a3660 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:30.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.584+0000 7fdfa4adf700 1 --2- 172.21.15.90:0/1647674777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfa0102820 0x7fdfa019e0b0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:30.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.584+0000 7fdfa4adf700 1 --2- 172.21.15.90:0/1647674777 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdfa00feeb0 0x7fdfa0196b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:30.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.584+0000 7fdfa4adf700 1 --2- 172.21.15.90:0/1647674777 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdfa0101e80 0x7fdfa0197090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:30.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.584+0000 7fdfa4adf700 1 -- 172.21.15.90:0/1647674777 >> 172.21.15.90:0/1647674777 conn(0x7fdfa00fa7e0 msgr2=0x7fdfa01052a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:30.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.584+0000 7fdfa4adf700 1 -- 172.21.15.90:0/1647674777 shutdown_connections 2024-02-21T23:46:30.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.584+0000 7fdfa4adf700 1 -- 172.21.15.90:0/1647674777 wait complete. 2024-02-21T23:46:30.570 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 13 2024-02-21T23:46:30.581 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:46:30.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.720+0000 7f8c6a8c1700 1 -- 172.21.15.90:0/4017317069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c640ffaf0 msgr2=0x7f8c640fff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:30.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.720+0000 7f8c6a8c1700 1 --2- 172.21.15.90:0/4017317069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c640ffaf0 0x7f8c640fff10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8c540099c0 tx=0x7f8c54009cd0 comp rx=0 tx=0).stop 2024-02-21T23:46:30.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.720+0000 7f8c6a8c1700 1 -- 172.21.15.90:0/4017317069 shutdown_connections 2024-02-21T23:46:30.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.720+0000 7f8c6a8c1700 1 --2- 172.21.15.90:0/4017317069 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8c64101680 0x7f8c64107f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:30.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.720+0000 7f8c6a8c1700 1 --2- 172.21.15.90:0/4017317069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c640ffaf0 0x7f8c640fff10 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:30.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.720+0000 7f8c6a8c1700 1 --2- 172.21.15.90:0/4017317069 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c64100ce0 0x7f8c64101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:30.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.720+0000 7f8c6a8c1700 1 -- 172.21.15.90:0/4017317069 >> 172.21.15.90:0/4017317069 conn(0x7f8c640fb060 msgr2=0x7f8c640fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:30.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.720+0000 7f8c6a8c1700 1 -- 172.21.15.90:0/4017317069 shutdown_connections 2024-02-21T23:46:30.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.720+0000 7f8c6a8c1700 1 -- 172.21.15.90:0/4017317069 wait complete. 2024-02-21T23:46:30.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.721+0000 7f8c6a8c1700 1 Processor -- start 2024-02-21T23:46:30.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.721+0000 7f8c6a8c1700 1 -- start start 2024-02-21T23:46:30.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.721+0000 7f8c6a8c1700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c640ffaf0 0x7f8c64068900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:30.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.721+0000 7f8c6a8c1700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8c64100ce0 0x7f8c6419b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:30.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.721+0000 7f8c6a8c1700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c64101680 0x7f8c6419de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:30.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.721+0000 7f8c6a8c1700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6419e5d0 con 0x7f8c64101680 2024-02-21T23:46:30.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.721+0000 7f8c6a8c1700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6419e740 con 0x7f8c640ffaf0 2024-02-21T23:46:30.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.721+0000 7f8c6a8c1700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6419e900 con 0x7f8c64100ce0 2024-02-21T23:46:30.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.721+0000 7f8c68e5e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c64101680 0x7f8c6419de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:30.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.721+0000 7f8c68e5e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c64101680 0x7f8c6419de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41546/0 (socket says 172.21.15.90:41546) 2024-02-21T23:46:30.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.721+0000 7f8c68e5e700 1 -- 172.21.15.90:0/2720165902 learned_addr learned my addr 172.21.15.90:0/2720165902 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:30.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.721+0000 7f8c637fe700 1 --2- 172.21.15.90:0/2720165902 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8c64100ce0 0x7f8c6419b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:30.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.721+0000 7f8c63fff700 1 --2- 172.21.15.90:0/2720165902 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c640ffaf0 0x7f8c64068900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:30.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.721+0000 7f8c68e5e700 1 -- 172.21.15.90:0/2720165902 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8c64100ce0 msgr2=0x7f8c6419b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:30.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.721+0000 7f8c68e5e700 1 --2- 172.21.15.90:0/2720165902 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8c64100ce0 0x7f8c6419b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:30.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.721+0000 7f8c68e5e700 1 -- 172.21.15.90:0/2720165902 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c640ffaf0 msgr2=0x7f8c64068900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:30.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.721+0000 7f8c68e5e700 1 --2- 172.21.15.90:0/2720165902 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c640ffaf0 0x7f8c64068900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:30.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.721+0000 7f8c68e5e700 1 -- 172.21.15.90:0/2720165902 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c540095f0 con 0x7f8c64101680 2024-02-21T23:46:30.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.721+0000 7f8c68e5e700 1 --2- 172.21.15.90:0/2720165902 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c64101680 0x7f8c6419de30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c007d70 tx=0x7f8c5c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:30.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.722+0000 7f8c617fa700 1 -- 172.21.15.90:0/2720165902 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c5c00ac90 con 0x7f8c64101680 2024-02-21T23:46:30.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.722+0000 7f8c6a8c1700 1 -- 172.21.15.90:0/2720165902 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c6419ebe0 con 0x7f8c64101680 2024-02-21T23:46:30.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.722+0000 7f8c617fa700 1 -- 172.21.15.90:0/2720165902 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8c5c00adf0 con 0x7f8c64101680 2024-02-21T23:46:30.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.722+0000 7f8c617fa700 1 -- 172.21.15.90:0/2720165902 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c5c0115e0 con 0x7f8c64101680 2024-02-21T23:46:30.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.722+0000 7f8c6a8c1700 1 -- 172.21.15.90:0/2720165902 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c6419f0a0 con 0x7f8c64101680 2024-02-21T23:46:30.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.723+0000 7f8c6a8c1700 1 -- 172.21.15.90:0/2720165902 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c64106ec0 con 0x7f8c64101680 2024-02-21T23:46:30.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.724+0000 7f8c617fa700 1 -- 172.21.15.90:0/2720165902 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7f8c5c004ad0 con 0x7f8c64101680 2024-02-21T23:46:30.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.724+0000 7f8c617fa700 1 --2- 172.21.15.90:0/2720165902 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f8c4c0a1120 0x7f8c4c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:30.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.724+0000 7f8c63fff700 1 --2- 172.21.15.90:0/2720165902 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f8c4c0a1120 0x7f8c4c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:30.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.724+0000 7f8c617fa700 1 -- 172.21.15.90:0/2720165902 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f8c5c0cca00 con 0x7f8c64101680 2024-02-21T23:46:30.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.725+0000 7f8c63fff700 1 --2- 172.21.15.90:0/2720165902 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f8c4c0a1120 0x7f8c4c0a35e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8c54009990 tx=0x7f8c54000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:30.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.732+0000 7f8c617fa700 1 -- 172.21.15.90:0/2720165902 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f8c5c096630 con 0x7f8c64101680 2024-02-21T23:46:30.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.942+0000 7f8c6a8c1700 1 -- 172.21.15.90:0/2720165902 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c640679a0 con 0x7f8c64101680 2024-02-21T23:46:30.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.943+0000 7f8c617fa700 1 -- 172.21.15.90:0/2720165902 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+18387 (secure 0 0 0) 0x7f8c5c099c50 con 0x7f8c64101680 2024-02-21T23:46:30.928 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:46:30.928 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":77,"fsid":"7ed81218-d112-11ee-95bd-87774f69a715","created":"2024-02-21T23:40:41.785526+0000","modified":"2024-02-21T23:46:01.867321+0000","last_up_change":"2024-02-21T23:46:00.860501+0000","last_in_change":"2024-02-21T23:45:44.423815+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:43:21.675789+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"77878890-d9a5-4d5b-a943-fec29ee0446e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6803","nonce":885319826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6805","nonce":885319826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6809","nonce":885319826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6807","nonce":885319826}]},"public_addr":"172.21.15.90:6803/885319826","cluster_addr":"172.21.15.90:6805/885319826","heartbeat_back_addr":"172.21.15.90:6809/885319826","heartbeat_front_addr":"172.21.15.90:6807/885319826","state":["exists","up"]},{"osd":1,"uuid":"7d6ea950-7a57-4f13-ab38-10a658912783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6801","nonce":4085240925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6803","nonce":4085240925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6807","nonce":4085240925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6805","nonce":4085240925}]},"public_addr":"172.21.15.176:6801/4085240925","cluster_addr":"172.21.15.176:6803/4085240925","heartbeat_back_addr":"172.21.15.176:6807/4085240925","heartbeat_front_addr":"172.21.15.176:6805/4085240925","state":["exists","up"]},{"osd":2,"uuid":"b83486bb-27f2-4d56-97fb-fb336fec4b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6801","nonce":3597250789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6803","nonce":3597250789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6807","nonce":3597250789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6805","nonce":3597250789}]},"public_addr":"172.21.15.178:6801/3597250789","cluster_addr":"172.21.15.178:6803/3597250789","heartbeat_back_addr":"172.21.15.178:6807/3597250789","heartbeat_front_addr":"172.21.15.178:6805/3597250789","state":["exists","up"]},{"osd":3,"uuid":"870e5bb6-eed5-4a0a-8e03-38de4f816b44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6811","nonce":3299043821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6813","nonce":3299043821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6817","nonce":3299043821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6815","nonce":3299043821}]},"public_addr":"172.21.15.90:6811/3299043821","cluster_addr":"172.21.15.90:6813/3299043821","heartbeat_back_addr":"172.21.15.90:6817/3299043821","heartbeat_front_addr":"172.21.15.90:6815/3299043821","state":["exists","up"]},{"osd":4,"uuid":"cb9f7f48-76e0-4368-ab87-2a09a8487ccd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6809","nonce":3548915687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6811","nonce":3548915687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6815","nonce":3548915687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6813","nonce":3548915687}]},"public_addr":"172.21.15.176:6809/3548915687","cluster_addr":"172.21.15.176:6811/3548915687","heartbeat_back_addr":"172.21.15.176:6815/3548915687","heartbeat_front_addr":"172.21.15.176:6813/3548915687","state":["exists","up"]},{"osd":5,"uuid":"54749022-c50d-4831-a1d6-8412b59c8a01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6809","nonce":286274518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6811","nonce":286274518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6815","nonce":286274518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6813","nonce":286274518}]},"public_addr":"172.21.15.178:6809/286274518","cluster_addr":"172.21.15.178:6811/286274518","heartbeat_back_addr":"172.21.15.178:6815/286274518","heartbeat_front_addr":"172.21.15.178:6813/286274518","state":["exists","up"]},{"osd":6,"uuid":"09bb5393-2efd-4a63-b0e8-99bef5bced13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6819","nonce":67640620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6821","nonce":67640620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6825","nonce":67640620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6823","nonce":67640620}]},"public_addr":"172.21.15.90:6819/67640620","cluster_addr":"172.21.15.90:6821/67640620","heartbeat_back_addr":"172.21.15.90:6825/67640620","heartbeat_front_addr":"172.21.15.90:6823/67640620","state":["exists","up"]},{"osd":7,"uuid":"7a859e69-6b63-467b-a3b3-99247d66f5c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6817","nonce":3998561598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6819","nonce":3998561598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6823","nonce":3998561598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6821","nonce":3998561598}]},"public_addr":"172.21.15.176:6817/3998561598","cluster_addr":"172.21.15.176:6819/3998561598","heartbeat_back_addr":"172.21.15.176:6823/3998561598","heartbeat_front_addr":"172.21.15.176:6821/3998561598","state":["exists","up"]},{"osd":8,"uuid":"bc23144e-22a5-4894-a8b5-9323383574d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6817","nonce":490229963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6819","nonce":490229963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6823","nonce":490229963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6821","nonce":490229963}]},"public_addr":"172.21.15.178:6817/490229963","cluster_addr":"172.21.15.178:6819/490229963","heartbeat_back_addr":"172.21.15.178:6823/490229963","heartbeat_front_addr":"172.21.15.178:6821/490229963","state":["exists","up"]},{"osd":9,"uuid":"0c2cecd4-55de-4fc4-a6f4-ca97c8460444","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6827","nonce":2061394761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6829","nonce":2061394761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6833","nonce":2061394761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6831","nonce":2061394761}]},"public_addr":"172.21.15.90:6827/2061394761","cluster_addr":"172.21.15.90:6829/2061394761","heartbeat_back_addr":"172.21.15.90:6833/2061394761","heartbeat_front_addr":"172.21.15.90:6831/2061394761","state":["exists","up"]},{"osd":10,"uuid":"fa04232b-b1ec-447a-8e94-7c9bf26d0291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6825","nonce":639411757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6827","nonce":639411757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6831","nonce":639411757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6829","nonce":639411757}]},"public_addr":"172.21.15.176:6825/639411757","cluster_addr":"172.21.15.176:6827/639411757","heartbeat_back_addr":"172.21.15.176:6831/639411757","heartbeat_front_addr":"172.21.15.176:6829/639411757","state":["exists","up"]},{"osd":11,"uuid":"7282e294-64fc-4856-9c74-86115f69408c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6825","nonce":4144536256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6827","nonce":4144536256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6831","nonce":4144536256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6829","nonce":4144536256}]},"public_addr":"172.21.15.178:6825/4144536256","cluster_addr":"172.21.15.178:6827/4144536256","heartbeat_back_addr":"172.21.15.178:6831/4144536256","heartbeat_front_addr":"172.21.15.178:6829/4144536256","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:42:44.280345+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:01.235630+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:19.408896+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:36.201749+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:53.401854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:44:11.342158+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:44:29.324502+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:44:46.821595+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:04.642344+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:23.019004+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:41.140149+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:59.401951+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/159655312":"2024-02-22T23:41:33.360021+0000","172.21.15.90:6801/1385551604":"2024-02-22T23:41:33.360021+0000","172.21.15.90:0/1452574770":"2024-02-22T23:41:33.360021+0000","172.21.15.90:0/3815292343":"2024-02-22T23:41:11.484145+0000","172.21.15.90:6800/1385551604":"2024-02-22T23:41:33.360021+0000","172.21.15.90:6801/4247695574":"2024-02-22T23:41:11.484145+0000","172.21.15.90:0/1328266622":"2024-02-22T23:41:11.484145+0000","172.21.15.90:6800/4247695574":"2024-02-22T23:41:11.484145+0000","172.21.15.90:0/2569421522":"2024-02-22T23:41:11.484145+0000","172.21.15.90:0/3019701027":"2024-02-22T23:41:33.360021+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:46:30.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.946+0000 7f8c6a8c1700 1 -- 172.21.15.90:0/2720165902 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f8c4c0a1120 msgr2=0x7f8c4c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:30.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.946+0000 7f8c6a8c1700 1 --2- 172.21.15.90:0/2720165902 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f8c4c0a1120 0x7f8c4c0a35e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8c54009990 tx=0x7f8c54000bc0 comp rx=0 tx=0).stop 2024-02-21T23:46:30.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.946+0000 7f8c6a8c1700 1 -- 172.21.15.90:0/2720165902 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c64101680 msgr2=0x7f8c6419de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:30.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.946+0000 7f8c6a8c1700 1 --2- 172.21.15.90:0/2720165902 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c64101680 0x7f8c6419de30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c007d70 tx=0x7f8c5c00a340 comp rx=0 tx=0).stop 2024-02-21T23:46:30.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.946+0000 7f8c6a8c1700 1 -- 172.21.15.90:0/2720165902 shutdown_connections 2024-02-21T23:46:30.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.946+0000 7f8c6a8c1700 1 --2- 172.21.15.90:0/2720165902 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f8c4c0a1120 0x7f8c4c0a35e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:30.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.946+0000 7f8c6a8c1700 1 --2- 172.21.15.90:0/2720165902 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c64101680 0x7f8c6419de30 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:30.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.946+0000 7f8c6a8c1700 1 --2- 172.21.15.90:0/2720165902 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c640ffaf0 0x7f8c64068900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:30.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.946+0000 7f8c6a8c1700 1 --2- 172.21.15.90:0/2720165902 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8c64100ce0 0x7f8c6419b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:30.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.946+0000 7f8c6a8c1700 1 -- 172.21.15.90:0/2720165902 >> 172.21.15.90:0/2720165902 conn(0x7f8c640fb060 msgr2=0x7f8c64104000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:30.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.946+0000 7f8c6a8c1700 1 -- 172.21.15.90:0/2720165902 shutdown_connections 2024-02-21T23:46:30.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:30.946+0000 7f8c6a8c1700 1 -- 172.21.15.90:0/2720165902 wait complete. 2024-02-21T23:46:30.944 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:43:21.675789+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:46:30.944 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:46:31.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.092+0000 7fd8b10d2700 1 -- 172.21.15.90:0/2761560433 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8ac1017a0 msgr2=0x7fd8ac108050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:31.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.092+0000 7fd8b10d2700 1 --2- 172.21.15.90:0/2761560433 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8ac1017a0 0x7fd8ac108050 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd8a400b0d0 tx=0x7fd8a400b3e0 comp rx=0 tx=0).stop 2024-02-21T23:46:31.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.092+0000 7fd8b10d2700 1 -- 172.21.15.90:0/2761560433 shutdown_connections 2024-02-21T23:46:31.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.092+0000 7fd8b10d2700 1 --2- 172.21.15.90:0/2761560433 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8ac1017a0 0x7fd8ac108050 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:31.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.092+0000 7fd8b10d2700 1 --2- 172.21.15.90:0/2761560433 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd8ac0ffc10 0x7fd8ac100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:31.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.092+0000 7fd8b10d2700 1 --2- 172.21.15.90:0/2761560433 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd8ac100e00 0x7fd8ac101260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:31.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.092+0000 7fd8b10d2700 1 -- 172.21.15.90:0/2761560433 >> 172.21.15.90:0/2761560433 conn(0x7fd8ac0fb180 msgr2=0x7fd8ac0fd5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:31.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.092+0000 7fd8b10d2700 1 -- 172.21.15.90:0/2761560433 shutdown_connections 2024-02-21T23:46:31.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.092+0000 7fd8b10d2700 1 -- 172.21.15.90:0/2761560433 wait complete. 2024-02-21T23:46:31.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.093+0000 7fd8b10d2700 1 Processor -- start 2024-02-21T23:46:31.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.093+0000 7fd8b10d2700 1 -- start start 2024-02-21T23:46:31.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.093+0000 7fd8b10d2700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd8ac0ffc10 0x7fd8ac103b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:31.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.093+0000 7fd8b10d2700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd8ac100e00 0x7fd8ac1021e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:31.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.093+0000 7fd8b10d2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8ac1017a0 0x7fd8ac102720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:31.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.093+0000 7fd8b10d2700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ac102ec0 con 0x7fd8ac1017a0 2024-02-21T23:46:31.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.093+0000 7fd8b10d2700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ac103030 con 0x7fd8ac100e00 2024-02-21T23:46:31.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.093+0000 7fd8b10d2700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ac1a0df0 con 0x7fd8ac0ffc10 2024-02-21T23:46:31.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.093+0000 7fd8ab59e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8ac1017a0 0x7fd8ac102720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:31.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.093+0000 7fd8ab59e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8ac1017a0 0x7fd8ac102720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41566/0 (socket says 172.21.15.90:41566) 2024-02-21T23:46:31.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.093+0000 7fd8ab59e700 1 -- 172.21.15.90:0/3111238275 learned_addr learned my addr 172.21.15.90:0/3111238275 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:31.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.093+0000 7fd8aad9d700 1 --2- 172.21.15.90:0/3111238275 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd8ac0ffc10 0x7fd8ac103b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:31.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.094+0000 7fd8aa59c700 1 --2- 172.21.15.90:0/3111238275 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd8ac100e00 0x7fd8ac1021e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:31.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.094+0000 7fd8ab59e700 1 -- 172.21.15.90:0/3111238275 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd8ac0ffc10 msgr2=0x7fd8ac103b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:31.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.094+0000 7fd8ab59e700 1 --2- 172.21.15.90:0/3111238275 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd8ac0ffc10 0x7fd8ac103b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:31.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.094+0000 7fd8ab59e700 1 -- 172.21.15.90:0/3111238275 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd8ac100e00 msgr2=0x7fd8ac1021e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:31.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.094+0000 7fd8ab59e700 1 --2- 172.21.15.90:0/3111238275 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd8ac100e00 0x7fd8ac1021e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:31.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.094+0000 7fd8ab59e700 1 -- 172.21.15.90:0/3111238275 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a4009d00 con 0x7fd8ac1017a0 2024-02-21T23:46:31.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.094+0000 7fd8ab59e700 1 --2- 172.21.15.90:0/3111238275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8ac1017a0 0x7fd8ac102720 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd8a4007b60 tx=0x7fd8a4008f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:31.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.094+0000 7fd893fff700 1 -- 172.21.15.90:0/3111238275 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a400e040 con 0x7fd8ac1017a0 2024-02-21T23:46:31.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.094+0000 7fd893fff700 1 -- 172.21.15.90:0/3111238275 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd8a401d930 con 0x7fd8ac1017a0 2024-02-21T23:46:31.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.094+0000 7fd8b10d2700 1 -- 172.21.15.90:0/3111238275 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8ac1a1020 con 0x7fd8ac1017a0 2024-02-21T23:46:31.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.094+0000 7fd893fff700 1 -- 172.21.15.90:0/3111238275 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a401ba00 con 0x7fd8ac1017a0 2024-02-21T23:46:31.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.094+0000 7fd8b10d2700 1 -- 172.21.15.90:0/3111238275 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8ac1a1440 con 0x7fd8ac1017a0 2024-02-21T23:46:31.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.095+0000 7fd8b10d2700 1 -- 172.21.15.90:0/3111238275 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8ac0671f0 con 0x7fd8ac1017a0 2024-02-21T23:46:31.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.096+0000 7fd893fff700 1 -- 172.21.15.90:0/3111238275 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7fd8a401daa0 con 0x7fd8ac1017a0 2024-02-21T23:46:31.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.097+0000 7fd893fff700 1 --2- 172.21.15.90:0/3111238275 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fd8940a1120 0x7fd8940a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:31.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.097+0000 7fd8aad9d700 1 --2- 172.21.15.90:0/3111238275 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fd8940a1120 0x7fd8940a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:31.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.097+0000 7fd893fff700 1 -- 172.21.15.90:0/3111238275 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd8a40cde50 con 0x7fd8ac1017a0 2024-02-21T23:46:31.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.097+0000 7fd8aad9d700 1 --2- 172.21.15.90:0/3111238275 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fd8940a1120 0x7fd8940a35e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac100c60 tx=0x7fd89c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:31.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.101+0000 7fd893fff700 1 -- 172.21.15.90:0/3111238275 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fd8a4097b10 con 0x7fd8ac1017a0 2024-02-21T23:46:31.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.337+0000 7fd8b10d2700 1 -- 172.21.15.90:0/3111238275 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd8ac1a1750 con 0x7fd8ac1017a0 2024-02-21T23:46:31.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.338+0000 7fd893fff700 1 -- 172.21.15.90:0/3111238275 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd8a409b130 con 0x7fd8ac1017a0 2024-02-21T23:46:31.322 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-02-21T23:46:31.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.340+0000 7fd8b10d2700 1 -- 172.21.15.90:0/3111238275 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fd8940a1120 msgr2=0x7fd8940a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:31.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.340+0000 7fd8b10d2700 1 --2- 172.21.15.90:0/3111238275 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fd8940a1120 0x7fd8940a35e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac100c60 tx=0x7fd89c009260 comp rx=0 tx=0).stop 2024-02-21T23:46:31.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.340+0000 7fd8b10d2700 1 -- 172.21.15.90:0/3111238275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8ac1017a0 msgr2=0x7fd8ac102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:31.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.340+0000 7fd8b10d2700 1 --2- 172.21.15.90:0/3111238275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8ac1017a0 0x7fd8ac102720 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd8a4007b60 tx=0x7fd8a4008f10 comp rx=0 tx=0).stop 2024-02-21T23:46:31.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.340+0000 7fd8b10d2700 1 -- 172.21.15.90:0/3111238275 shutdown_connections 2024-02-21T23:46:31.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.340+0000 7fd8b10d2700 1 --2- 172.21.15.90:0/3111238275 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fd8940a1120 0x7fd8940a35e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:31.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.340+0000 7fd8b10d2700 1 --2- 172.21.15.90:0/3111238275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8ac1017a0 0x7fd8ac102720 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:31.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.340+0000 7fd8b10d2700 1 --2- 172.21.15.90:0/3111238275 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd8ac0ffc10 0x7fd8ac103b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:31.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.340+0000 7fd8b10d2700 1 --2- 172.21.15.90:0/3111238275 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd8ac100e00 0x7fd8ac1021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:31.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.340+0000 7fd8b10d2700 1 -- 172.21.15.90:0/3111238275 >> 172.21.15.90:0/3111238275 conn(0x7fd8ac0fb180 msgr2=0x7fd8ac0fcc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:31.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.340+0000 7fd8b10d2700 1 -- 172.21.15.90:0/3111238275 shutdown_connections 2024-02-21T23:46:31.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.340+0000 7fd8b10d2700 1 -- 172.21.15.90:0/3111238275 wait complete. 2024-02-21T23:46:31.340 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-21T23:46:31.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.485+0000 7fda8f997700 1 -- 172.21.15.90:0/1423179661 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda881068f0 msgr2=0x7fda88108d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:31.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.485+0000 7fda8f997700 1 --2- 172.21.15.90:0/1423179661 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda881068f0 0x7fda88108d60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fda8000b0a0 tx=0x7fda8000b3b0 comp rx=0 tx=0).stop 2024-02-21T23:46:31.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.485+0000 7fda8f997700 1 -- 172.21.15.90:0/1423179661 shutdown_connections 2024-02-21T23:46:31.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.485+0000 7fda8f997700 1 --2- 172.21.15.90:0/1423179661 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda881068f0 0x7fda88108d60 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:31.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.485+0000 7fda8f997700 1 --2- 172.21.15.90:0/1423179661 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fda88101690 0x7fda88103a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:31.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.485+0000 7fda8f997700 1 --2- 172.21.15.90:0/1423179661 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda88103fc0 0x7fda881063b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:31.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.485+0000 7fda8f997700 1 -- 172.21.15.90:0/1423179661 >> 172.21.15.90:0/1423179661 conn(0x7fda880fb0a0 msgr2=0x7fda880fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:31.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.485+0000 7fda8f997700 1 -- 172.21.15.90:0/1423179661 shutdown_connections 2024-02-21T23:46:31.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.485+0000 7fda8f997700 1 -- 172.21.15.90:0/1423179661 wait complete. 2024-02-21T23:46:31.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.485+0000 7fda8f997700 1 Processor -- start 2024-02-21T23:46:31.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.485+0000 7fda8f997700 1 -- start start 2024-02-21T23:46:31.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.485+0000 7fda8f997700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda88101690 0x7fda88100fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:31.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.485+0000 7fda8f997700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda88103fc0 0x7fda880ff640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:31.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.485+0000 7fda8f997700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fda881068f0 0x7fda880ffb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:31.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.485+0000 7fda8f997700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda88100320 con 0x7fda88101690 2024-02-21T23:46:31.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.485+0000 7fda8f997700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda88100490 con 0x7fda88103fc0 2024-02-21T23:46:31.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.485+0000 7fda8f997700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda881a0c90 con 0x7fda881068f0 2024-02-21T23:46:31.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.486+0000 7fda8d733700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda88101690 0x7fda88100fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:31.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.486+0000 7fda8d733700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda88101690 0x7fda88100fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41586/0 (socket says 172.21.15.90:41586) 2024-02-21T23:46:31.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.486+0000 7fda8d733700 1 -- 172.21.15.90:0/1850955331 learned_addr learned my addr 172.21.15.90:0/1850955331 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:31.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.486+0000 7fda8df34700 1 --2- 172.21.15.90:0/1850955331 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fda881068f0 0x7fda880ffb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:31.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.486+0000 7fda8d733700 1 -- 172.21.15.90:0/1850955331 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fda881068f0 msgr2=0x7fda880ffb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:31.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.486+0000 7fda8d733700 1 --2- 172.21.15.90:0/1850955331 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fda881068f0 0x7fda880ffb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:31.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.486+0000 7fda8d733700 1 -- 172.21.15.90:0/1850955331 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda88103fc0 msgr2=0x7fda880ff640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:31.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.486+0000 7fda8cf32700 1 --2- 172.21.15.90:0/1850955331 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda88103fc0 0x7fda880ff640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:31.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.486+0000 7fda8d733700 1 --2- 172.21.15.90:0/1850955331 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda88103fc0 0x7fda880ff640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:31.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.486+0000 7fda8d733700 1 -- 172.21.15.90:0/1850955331 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda80009d00 con 0x7fda88101690 2024-02-21T23:46:31.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.486+0000 7fda8d733700 1 --2- 172.21.15.90:0/1850955331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda88101690 0x7fda88100fb0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fda8400d980 tx=0x7fda8400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:31.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.486+0000 7fda7e7fc700 1 -- 172.21.15.90:0/1850955331 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda84004220 con 0x7fda88101690 2024-02-21T23:46:31.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.486+0000 7fda7e7fc700 1 -- 172.21.15.90:0/1850955331 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fda84007640 con 0x7fda88101690 2024-02-21T23:46:31.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.486+0000 7fda8f997700 1 -- 172.21.15.90:0/1850955331 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda881a0f20 con 0x7fda88101690 2024-02-21T23:46:31.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.486+0000 7fda7e7fc700 1 -- 172.21.15.90:0/1850955331 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda84004e20 con 0x7fda88101690 2024-02-21T23:46:31.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.486+0000 7fda8f997700 1 -- 172.21.15.90:0/1850955331 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda881a1340 con 0x7fda88101690 2024-02-21T23:46:31.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.487+0000 7fda8f997700 1 -- 172.21.15.90:0/1850955331 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda880fcc20 con 0x7fda88101690 2024-02-21T23:46:31.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.488+0000 7fda7e7fc700 1 -- 172.21.15.90:0/1850955331 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7fda8400b900 con 0x7fda88101690 2024-02-21T23:46:31.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.489+0000 7fda7e7fc700 1 --2- 172.21.15.90:0/1850955331 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fda740a1120 0x7fda740a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:31.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.489+0000 7fda7e7fc700 1 -- 172.21.15.90:0/1850955331 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fda840cc330 con 0x7fda88101690 2024-02-21T23:46:31.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.489+0000 7fda8cf32700 1 --2- 172.21.15.90:0/1850955331 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fda740a1120 0x7fda740a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:31.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.489+0000 7fda8cf32700 1 --2- 172.21.15.90:0/1850955331 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fda740a1120 0x7fda740a35e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fda78005950 tx=0x7fda78005c70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:31.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.493+0000 7fda7e7fc700 1 -- 172.21.15.90:0/1850955331 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fda84095f60 con 0x7fda88101690 2024-02-21T23:46:31.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:31 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:31.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:31 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:31.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:31 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:31.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:31 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:31.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:31 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:31.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:31 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:31.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:31 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/1647674777' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-21T23:46:31.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:31 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:31.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:31 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:46:31.585 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:31 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:31.585 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:31 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:46:31.585 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:31 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/2720165902' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-21T23:46:31.585 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:31 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/3111238275' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-21T23:46:31.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:31 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:31.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:31 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:31.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:31 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:31.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:31 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:31.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:31 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:31.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:31 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:31.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:31 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/1647674777' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-21T23:46:31.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:31 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:31.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:31 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:46:31.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:31 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:31.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:31 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:46:31.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:31 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/2720165902' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-21T23:46:31.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:31 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/3111238275' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-21T23:46:31.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.718+0000 7fda8f997700 1 -- 172.21.15.90:0/1850955331 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fda8802cc70 con 0x7fda88101690 2024-02-21T23:46:31.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.719+0000 7fda7e7fc700 1 -- 172.21.15.90:0/1850955331 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v77) v1 ==== 84+0+65 (secure 0 0 0) 0x7fda84099580 con 0x7fda88101690 2024-02-21T23:46:31.703 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:46:31.703 INFO:teuthology.orchestra.run.smithi090.stdout:[ 2024-02-21T23:46:31.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-21T23:46:31.704 INFO:teuthology.orchestra.run.smithi090.stdout: "poolnum": 1, 2024-02-21T23:46:31.704 INFO:teuthology.orchestra.run.smithi090.stdout: "poolname": ".mgr" 2024-02-21T23:46:31.704 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-21T23:46:31.704 INFO:teuthology.orchestra.run.smithi090.stdout:] 2024-02-21T23:46:31.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.721+0000 7fda8f997700 1 -- 172.21.15.90:0/1850955331 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fda740a1120 msgr2=0x7fda740a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:31.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.721+0000 7fda8f997700 1 --2- 172.21.15.90:0/1850955331 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fda740a1120 0x7fda740a35e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fda78005950 tx=0x7fda78005c70 comp rx=0 tx=0).stop 2024-02-21T23:46:31.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.721+0000 7fda8f997700 1 -- 172.21.15.90:0/1850955331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda88101690 msgr2=0x7fda88100fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:31.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.721+0000 7fda8f997700 1 --2- 172.21.15.90:0/1850955331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda88101690 0x7fda88100fb0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fda8400d980 tx=0x7fda8400dd40 comp rx=0 tx=0).stop 2024-02-21T23:46:31.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.721+0000 7fda8f997700 1 -- 172.21.15.90:0/1850955331 shutdown_connections 2024-02-21T23:46:31.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.721+0000 7fda8f997700 1 --2- 172.21.15.90:0/1850955331 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fda740a1120 0x7fda740a35e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:31.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.721+0000 7fda8f997700 1 --2- 172.21.15.90:0/1850955331 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fda881068f0 0x7fda880ffb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:31.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.721+0000 7fda8f997700 1 --2- 172.21.15.90:0/1850955331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda88101690 0x7fda88100fb0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:31.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.721+0000 7fda8f997700 1 --2- 172.21.15.90:0/1850955331 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda88103fc0 0x7fda880ff640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:31.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.721+0000 7fda8f997700 1 -- 172.21.15.90:0/1850955331 >> 172.21.15.90:0/1850955331 conn(0x7fda880fb0a0 msgr2=0x7fda88101ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:31.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.721+0000 7fda8f997700 1 -- 172.21.15.90:0/1850955331 shutdown_connections 2024-02-21T23:46:31.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.721+0000 7fda8f997700 1 -- 172.21.15.90:0/1850955331 wait complete. 2024-02-21T23:46:31.718 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:31 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:31.718 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:31 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:31.718 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:31 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:31.718 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:31 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:31.718 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:31 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:31.718 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:31 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:31.718 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:31 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/1647674777' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-21T23:46:31.718 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:31 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:31.718 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:31 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:46:31.719 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:31 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:31.719 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:31 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-21T23:46:31.719 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:31 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/2720165902' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-21T23:46:31.719 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:31 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/3111238275' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-21T23:46:31.720 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-02-21T23:46:31.720 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-02-21T23:46:31.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.877+0000 7f4b16b46700 1 -- 172.21.15.90:0/3562218417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b100737d0 msgr2=0x7f4b10073c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:31.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.877+0000 7f4b16b46700 1 --2- 172.21.15.90:0/3562218417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b100737d0 0x7f4b10073c50 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4afc009990 tx=0x7f4afc009ca0 comp rx=0 tx=0).stop 2024-02-21T23:46:31.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.877+0000 7f4b16b46700 1 -- 172.21.15.90:0/3562218417 shutdown_connections 2024-02-21T23:46:31.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.877+0000 7f4b16b46700 1 --2- 172.21.15.90:0/3562218417 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b10108bc0 0x7f4b1010b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:31.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.877+0000 7f4b16b46700 1 --2- 172.21.15.90:0/3562218417 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4b10074db0 0x7f4b10073200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:31.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.877+0000 7f4b16b46700 1 --2- 172.21.15.90:0/3562218417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b100737d0 0x7f4b10073c50 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:31.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.877+0000 7f4b16b46700 1 -- 172.21.15.90:0/3562218417 >> 172.21.15.90:0/3562218417 conn(0x7f4b100fc270 msgr2=0x7f4b100fe6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:31.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.877+0000 7f4b16b46700 1 -- 172.21.15.90:0/3562218417 shutdown_connections 2024-02-21T23:46:31.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.877+0000 7f4b16b46700 1 -- 172.21.15.90:0/3562218417 wait complete. 2024-02-21T23:46:31.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.877+0000 7f4b16b46700 1 Processor -- start 2024-02-21T23:46:31.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.877+0000 7f4b16b46700 1 -- start start 2024-02-21T23:46:31.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.878+0000 7f4b16b46700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b100737d0 0x7f4b10198eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:31.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.878+0000 7f4b16b46700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4b10074db0 0x7f4b101993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:31.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.878+0000 7f4b16b46700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b10108bc0 0x7f4b101a0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:31.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.878+0000 7f4b16b46700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b101a09e0 con 0x7f4b10108bc0 2024-02-21T23:46:31.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.878+0000 7f4b16b46700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b101a0b20 con 0x7f4b100737d0 2024-02-21T23:46:31.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.878+0000 7f4b16b46700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b101a0cb0 con 0x7f4b10074db0 2024-02-21T23:46:31.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.878+0000 7f4b150e3700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b10108bc0 0x7f4b101a0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:31.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.878+0000 7f4b148e2700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b100737d0 0x7f4b10198eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:31.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.878+0000 7f4b150e3700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b10108bc0 0x7f4b101a0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41594/0 (socket says 172.21.15.90:41594) 2024-02-21T23:46:31.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.878+0000 7f4b150e3700 1 -- 172.21.15.90:0/799478952 learned_addr learned my addr 172.21.15.90:0/799478952 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:31.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.878+0000 7f4b0ffff700 1 --2- 172.21.15.90:0/799478952 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4b10074db0 0x7f4b101993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:31.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.878+0000 7f4b150e3700 1 -- 172.21.15.90:0/799478952 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4b10074db0 msgr2=0x7f4b101993f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:31.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.878+0000 7f4b150e3700 1 --2- 172.21.15.90:0/799478952 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4b10074db0 0x7f4b101993f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:31.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.878+0000 7f4b150e3700 1 -- 172.21.15.90:0/799478952 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b100737d0 msgr2=0x7f4b10198eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:31.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.878+0000 7f4b150e3700 1 --2- 172.21.15.90:0/799478952 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b100737d0 0x7f4b10198eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:31.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.878+0000 7f4b150e3700 1 -- 172.21.15.90:0/799478952 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4afc0095f0 con 0x7f4b10108bc0 2024-02-21T23:46:31.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.878+0000 7f4b150e3700 1 --2- 172.21.15.90:0/799478952 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b10108bc0 0x7f4b101a0410 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4b0800ef30 tx=0x7f4b0800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:31.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.878+0000 7f4b0dffb700 1 -- 172.21.15.90:0/799478952 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b08018070 con 0x7f4b10108bc0 2024-02-21T23:46:31.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.878+0000 7f4b16b46700 1 -- 172.21.15.90:0/799478952 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b101a0f30 con 0x7f4b10108bc0 2024-02-21T23:46:31.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.878+0000 7f4b0dffb700 1 -- 172.21.15.90:0/799478952 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4b0800f040 con 0x7f4b10108bc0 2024-02-21T23:46:31.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.878+0000 7f4b0dffb700 1 -- 172.21.15.90:0/799478952 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0800cd00 con 0x7f4b10108bc0 2024-02-21T23:46:31.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.879+0000 7f4b16b46700 1 -- 172.21.15.90:0/799478952 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b101a13b0 con 0x7f4b10108bc0 2024-02-21T23:46:31.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.880+0000 7f4b16b46700 1 -- 172.21.15.90:0/799478952 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b10107980 con 0x7f4b10108bc0 2024-02-21T23:46:31.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.881+0000 7f4b0dffb700 1 -- 172.21.15.90:0/799478952 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7f4b08003a00 con 0x7f4b10108bc0 2024-02-21T23:46:31.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.881+0000 7f4b0dffb700 1 --2- 172.21.15.90:0/799478952 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f4b000a1120 0x7f4b000a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:31.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.881+0000 7f4b0dffb700 1 -- 172.21.15.90:0/799478952 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4b080cbf80 con 0x7f4b10108bc0 2024-02-21T23:46:31.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.881+0000 7f4b148e2700 1 --2- 172.21.15.90:0/799478952 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f4b000a1120 0x7f4b000a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:31.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.882+0000 7f4b148e2700 1 --2- 172.21.15.90:0/799478952 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f4b000a1120 0x7f4b000a35e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4b04009c60 tx=0x7f4b04009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:31.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:31.886+0000 7f4b0dffb700 1 -- 172.21.15.90:0/799478952 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f4b080ff960 con 0x7f4b10108bc0 2024-02-21T23:46:32.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:32.106+0000 7f4b16b46700 1 -- 172.21.15.90:0/799478952 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f4b10067ed0 con 0x7f4b10108bc0 2024-02-21T23:46:32.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:32 smithi178 ceph-mon[47547]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:32.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:32 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/1850955331' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-02-21T23:46:32.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:32 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:32.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:32 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:32.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:32 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:32.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:32 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:32.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:32 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:32.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:32 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:32.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:32 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:32.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:32 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:46:32.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:32 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:32.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:32 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/799478952' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-02-21T23:46:32.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:32 smithi090 ceph-mon[43385]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:32.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:32 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/1850955331' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-02-21T23:46:32.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:32 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:32.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:32 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:32.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:32 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:32.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:32 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:32.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:32 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:32.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:32 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:32.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:32 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:32.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:32 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:46:32.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:32 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:32.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:32 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/799478952' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-02-21T23:46:32.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:32 smithi176 ceph-mon[47172]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:32.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:32 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/1850955331' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-02-21T23:46:32.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:32 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:32.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:32 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:32.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:32 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:32.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:32 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:32.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:32 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:32.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:32 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:32.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:32 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-21T23:46:32.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:32 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-21T23:46:32.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:32 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:32.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:32 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/799478952' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-02-21T23:46:33.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.034+0000 7f4b0dffb700 1 -- 172.21.15.90:0/799478952 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v78) v1 ==== 138+0+0 (secure 0 0 0) 0x7f4b10067ed0 con 0x7f4b10108bc0 2024-02-21T23:46:33.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.038+0000 7f4b16b46700 1 -- 172.21.15.90:0/799478952 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f4b000a1120 msgr2=0x7f4b000a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:33.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.038+0000 7f4b16b46700 1 --2- 172.21.15.90:0/799478952 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f4b000a1120 0x7f4b000a35e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4b04009c60 tx=0x7f4b04009260 comp rx=0 tx=0).stop 2024-02-21T23:46:33.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.038+0000 7f4b16b46700 1 -- 172.21.15.90:0/799478952 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b10108bc0 msgr2=0x7f4b101a0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:33.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.038+0000 7f4b16b46700 1 --2- 172.21.15.90:0/799478952 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b10108bc0 0x7f4b101a0410 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4b0800ef30 tx=0x7f4b0800c3a0 comp rx=0 tx=0).stop 2024-02-21T23:46:33.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.038+0000 7f4b16b46700 1 -- 172.21.15.90:0/799478952 shutdown_connections 2024-02-21T23:46:33.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.038+0000 7f4b16b46700 1 --2- 172.21.15.90:0/799478952 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f4b000a1120 0x7f4b000a35e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:33.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.038+0000 7f4b16b46700 1 --2- 172.21.15.90:0/799478952 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b10108bc0 0x7f4b101a0410 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:33.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.038+0000 7f4b16b46700 1 --2- 172.21.15.90:0/799478952 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b100737d0 0x7f4b10198eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:33.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.038+0000 7f4b16b46700 1 --2- 172.21.15.90:0/799478952 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4b10074db0 0x7f4b101993f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:33.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.038+0000 7f4b16b46700 1 -- 172.21.15.90:0/799478952 >> 172.21.15.90:0/799478952 conn(0x7f4b100fc270 msgr2=0x7f4b10104ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:33.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.038+0000 7f4b16b46700 1 -- 172.21.15.90:0/799478952 shutdown_connections 2024-02-21T23:46:33.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.038+0000 7f4b16b46700 1 -- 172.21.15.90:0/799478952 wait complete. 2024-02-21T23:46:33.024 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_metadata' created 2024-02-21T23:46:33.037 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-02-21T23:46:33.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.186+0000 7f810185d700 1 -- 172.21.15.90:0/1376603924 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80fc101ef0 msgr2=0x7f80fc102350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:33.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.186+0000 7f810185d700 1 --2- 172.21.15.90:0/1376603924 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80fc101ef0 0x7f80fc102350 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f80f00099c0 tx=0x7f80f0009cd0 comp rx=0 tx=0).stop 2024-02-21T23:46:33.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.186+0000 7f810185d700 1 -- 172.21.15.90:0/1376603924 shutdown_connections 2024-02-21T23:46:33.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.187+0000 7f810185d700 1 --2- 172.21.15.90:0/1376603924 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80fc102890 0x7f80fc109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:33.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.187+0000 7f810185d700 1 --2- 172.21.15.90:0/1376603924 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f80fc100d00 0x7f80fc101120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:33.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.187+0000 7f810185d700 1 --2- 172.21.15.90:0/1376603924 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80fc101ef0 0x7f80fc102350 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:33.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.187+0000 7f810185d700 1 -- 172.21.15.90:0/1376603924 >> 172.21.15.90:0/1376603924 conn(0x7f80fc0fc270 msgr2=0x7f80fc0fe6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:33.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.187+0000 7f810185d700 1 -- 172.21.15.90:0/1376603924 shutdown_connections 2024-02-21T23:46:33.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.187+0000 7f810185d700 1 -- 172.21.15.90:0/1376603924 wait complete. 2024-02-21T23:46:33.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.187+0000 7f810185d700 1 Processor -- start 2024-02-21T23:46:33.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.187+0000 7f810185d700 1 -- start start 2024-02-21T23:46:33.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.187+0000 7f810185d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80fc100d00 0x7f80fc196c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:33.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.187+0000 7f810185d700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80fc101ef0 0x7f80fc1971d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:33.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.187+0000 7f810185d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f80fc102890 0x7f80fc19e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:33.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.187+0000 7f810185d700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc19e780 con 0x7f80fc100d00 2024-02-21T23:46:33.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.187+0000 7f810185d700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc19e8f0 con 0x7f80fc102890 2024-02-21T23:46:33.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.187+0000 7f810185d700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc19eab0 con 0x7f80fc101ef0 2024-02-21T23:46:33.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.188+0000 7f80faffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80fc100d00 0x7f80fc196c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:33.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.188+0000 7f80faffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80fc100d00 0x7f80fc196c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41612/0 (socket says 172.21.15.90:41612) 2024-02-21T23:46:33.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.188+0000 7f80faffd700 1 -- 172.21.15.90:0/4236800159 learned_addr learned my addr 172.21.15.90:0/4236800159 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:33.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.188+0000 7f80fb7fe700 1 --2- 172.21.15.90:0/4236800159 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f80fc102890 0x7f80fc19e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:33.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.188+0000 7f80fa7fc700 1 --2- 172.21.15.90:0/4236800159 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80fc101ef0 0x7f80fc1971d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:33.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.188+0000 7f80faffd700 1 -- 172.21.15.90:0/4236800159 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80fc101ef0 msgr2=0x7f80fc1971d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:33.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.188+0000 7f80faffd700 1 --2- 172.21.15.90:0/4236800159 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80fc101ef0 0x7f80fc1971d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:33.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.188+0000 7f80faffd700 1 -- 172.21.15.90:0/4236800159 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f80fc102890 msgr2=0x7f80fc19e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:33.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.188+0000 7f80faffd700 1 --2- 172.21.15.90:0/4236800159 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f80fc102890 0x7f80fc19e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:33.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.188+0000 7f80faffd700 1 -- 172.21.15.90:0/4236800159 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f00095f0 con 0x7f80fc100d00 2024-02-21T23:46:33.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.188+0000 7f80fa7fc700 1 --2- 172.21.15.90:0/4236800159 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80fc101ef0 0x7f80fc1971d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:46:33.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.188+0000 7f80faffd700 1 --2- 172.21.15.90:0/4236800159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80fc100d00 0x7f80fc196c90 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f80ec00d980 tx=0x7f80ec00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:33.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.188+0000 7f810085b700 1 -- 172.21.15.90:0/4236800159 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ec004020 con 0x7f80fc100d00 2024-02-21T23:46:33.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.188+0000 7f810185d700 1 -- 172.21.15.90:0/4236800159 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80fc19ed90 con 0x7f80fc100d00 2024-02-21T23:46:33.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.188+0000 7f810085b700 1 -- 172.21.15.90:0/4236800159 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f80ec009730 con 0x7f80fc100d00 2024-02-21T23:46:33.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.188+0000 7f810085b700 1 -- 172.21.15.90:0/4236800159 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ec003a30 con 0x7f80fc100d00 2024-02-21T23:46:33.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.188+0000 7f810185d700 1 -- 172.21.15.90:0/4236800159 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80fc19f200 con 0x7f80fc100d00 2024-02-21T23:46:33.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.189+0000 7f810185d700 1 -- 172.21.15.90:0/4236800159 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80fc1081d0 con 0x7f80fc100d00 2024-02-21T23:46:33.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.190+0000 7f810085b700 1 -- 172.21.15.90:0/4236800159 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7f80ec0098a0 con 0x7f80fc100d00 2024-02-21T23:46:33.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.191+0000 7f810085b700 1 --2- 172.21.15.90:0/4236800159 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f80e40a1120 0x7f80e40a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:33.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.191+0000 7f810085b700 1 -- 172.21.15.90:0/4236800159 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f80ec0cc5b0 con 0x7f80fc100d00 2024-02-21T23:46:33.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.191+0000 7f80fa7fc700 1 --2- 172.21.15.90:0/4236800159 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f80e40a1120 0x7f80e40a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:33.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.191+0000 7f80fa7fc700 1 --2- 172.21.15.90:0/4236800159 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f80e40a1120 0x7f80e40a35e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f80f000b580 tx=0x7f80f001a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:33.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.195+0000 7f810085b700 1 -- 172.21.15.90:0/4236800159 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f80ec096070 con 0x7f80fc100d00 2024-02-21T23:46:33.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:33.390+0000 7f810185d700 1 -- 172.21.15.90:0/4236800159 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f80fc067ed0 con 0x7f80fc100d00 2024-02-21T23:46:34.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.041+0000 7f810085b700 1 -- 172.21.15.90:0/4236800159 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v79) v1 ==== 171+0+0 (secure 0 0 0) 0x7f80ec099690 con 0x7f80fc100d00 2024-02-21T23:46:34.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.045+0000 7f810185d700 1 -- 172.21.15.90:0/4236800159 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f80e40a1120 msgr2=0x7f80e40a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:34.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.045+0000 7f810185d700 1 --2- 172.21.15.90:0/4236800159 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f80e40a1120 0x7f80e40a35e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f80f000b580 tx=0x7f80f001a040 comp rx=0 tx=0).stop 2024-02-21T23:46:34.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.045+0000 7f810185d700 1 -- 172.21.15.90:0/4236800159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80fc100d00 msgr2=0x7f80fc196c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:34.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.045+0000 7f810185d700 1 --2- 172.21.15.90:0/4236800159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80fc100d00 0x7f80fc196c90 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f80ec00d980 tx=0x7f80ec00dc90 comp rx=0 tx=0).stop 2024-02-21T23:46:34.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.046+0000 7f810185d700 1 -- 172.21.15.90:0/4236800159 shutdown_connections 2024-02-21T23:46:34.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.046+0000 7f810185d700 1 --2- 172.21.15.90:0/4236800159 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f80e40a1120 0x7f80e40a35e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:34.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.046+0000 7f810185d700 1 --2- 172.21.15.90:0/4236800159 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f80fc102890 0x7f80fc19e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:34.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.046+0000 7f810185d700 1 --2- 172.21.15.90:0/4236800159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80fc100d00 0x7f80fc196c90 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:34.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.046+0000 7f810185d700 1 --2- 172.21.15.90:0/4236800159 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80fc101ef0 0x7f80fc1971d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:34.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.046+0000 7f810185d700 1 -- 172.21.15.90:0/4236800159 >> 172.21.15.90:0/4236800159 conn(0x7f80fc0fc270 msgr2=0x7f80fc105310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:34.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.046+0000 7f810185d700 1 -- 172.21.15.90:0/4236800159 shutdown_connections 2024-02-21T23:46:34.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.047+0000 7f810185d700 1 -- 172.21.15.90:0/4236800159 wait complete. 2024-02-21T23:46:34.040 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_data' created 2024-02-21T23:46:34.051 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-02-21T23:46:34.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.191+0000 7f0ef3fff700 1 -- 172.21.15.90:0/1716277711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ef41003e0 msgr2=0x7f0ef4100840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:34.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.191+0000 7f0ef3fff700 1 --2- 172.21.15.90:0/1716277711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ef41003e0 0x7f0ef4100840 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0ee80099c0 tx=0x7f0ee8009cd0 comp rx=0 tx=0).stop 2024-02-21T23:46:34.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.191+0000 7f0ef3fff700 1 -- 172.21.15.90:0/1716277711 shutdown_connections 2024-02-21T23:46:34.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.191+0000 7f0ef3fff700 1 --2- 172.21.15.90:0/1716277711 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0ef4100d80 0x7f0ef4107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:34.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.191+0000 7f0ef3fff700 1 --2- 172.21.15.90:0/1716277711 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0ef40ff1f0 0x7f0ef40ff610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:34.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.191+0000 7f0ef3fff700 1 --2- 172.21.15.90:0/1716277711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ef41003e0 0x7f0ef4100840 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:34.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.191+0000 7f0ef3fff700 1 -- 172.21.15.90:0/1716277711 >> 172.21.15.90:0/1716277711 conn(0x7f0ef40fa760 msgr2=0x7f0ef40fcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:34.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.191+0000 7f0ef3fff700 1 -- 172.21.15.90:0/1716277711 shutdown_connections 2024-02-21T23:46:34.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.191+0000 7f0ef3fff700 1 -- 172.21.15.90:0/1716277711 wait complete. 2024-02-21T23:46:34.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.192+0000 7f0ef3fff700 1 Processor -- start 2024-02-21T23:46:34.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.192+0000 7f0ef3fff700 1 -- start start 2024-02-21T23:46:34.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.192+0000 7f0ef3fff700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0ef40ff1f0 0x7f0ef410c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:34.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.192+0000 7f0ef3fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0ef41003e0 0x7f0ef419b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:34.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.192+0000 7f0ef3fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ef4100d80 0x7f0ef419de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:34.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.192+0000 7f0ef3fff700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef419e5c0 con 0x7f0ef4100d80 2024-02-21T23:46:34.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.192+0000 7f0ef3fff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef419e730 con 0x7f0ef41003e0 2024-02-21T23:46:34.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.192+0000 7f0ef3fff700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef419e8f0 con 0x7f0ef40ff1f0 2024-02-21T23:46:34.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.192+0000 7f0ef37fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ef4100d80 0x7f0ef419de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:34.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.192+0000 7f0ef37fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ef4100d80 0x7f0ef419de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41640/0 (socket says 172.21.15.90:41640) 2024-02-21T23:46:34.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.192+0000 7f0ef37fe700 1 -- 172.21.15.90:0/3188262850 learned_addr learned my addr 172.21.15.90:0/3188262850 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:34.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.192+0000 7f0ef27fc700 1 --2- 172.21.15.90:0/3188262850 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0ef41003e0 0x7f0ef419b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:34.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.192+0000 7f0ef2ffd700 1 --2- 172.21.15.90:0/3188262850 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0ef40ff1f0 0x7f0ef410c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:34.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.193+0000 7f0ef37fe700 1 -- 172.21.15.90:0/3188262850 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0ef40ff1f0 msgr2=0x7f0ef410c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:34.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.193+0000 7f0ef37fe700 1 --2- 172.21.15.90:0/3188262850 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0ef40ff1f0 0x7f0ef410c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:34.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.193+0000 7f0ef37fe700 1 -- 172.21.15.90:0/3188262850 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0ef41003e0 msgr2=0x7f0ef419b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:34.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.193+0000 7f0ef37fe700 1 --2- 172.21.15.90:0/3188262850 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0ef41003e0 0x7f0ef419b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:34.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.193+0000 7f0ef37fe700 1 -- 172.21.15.90:0/3188262850 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee80095f0 con 0x7f0ef4100d80 2024-02-21T23:46:34.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.193+0000 7f0ef37fe700 1 --2- 172.21.15.90:0/3188262850 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ef4100d80 0x7f0ef419de20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0eec00ef30 tx=0x7f0eec00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:34.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.193+0000 7f0edbfff700 1 -- 172.21.15.90:0/3188262850 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eec018070 con 0x7f0ef4100d80 2024-02-21T23:46:34.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.193+0000 7f0ef3fff700 1 -- 172.21.15.90:0/3188262850 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ef419eb70 con 0x7f0ef4100d80 2024-02-21T23:46:34.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.193+0000 7f0edbfff700 1 -- 172.21.15.90:0/3188262850 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0eec00f040 con 0x7f0ef4100d80 2024-02-21T23:46:34.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.193+0000 7f0edbfff700 1 -- 172.21.15.90:0/3188262850 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eec00cd00 con 0x7f0ef4100d80 2024-02-21T23:46:34.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.193+0000 7f0ef3fff700 1 -- 172.21.15.90:0/3188262850 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ef419f080 con 0x7f0ef4100d80 2024-02-21T23:46:34.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.194+0000 7f0ef3fff700 1 -- 172.21.15.90:0/3188262850 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ef4105dc0 con 0x7f0ef4100d80 2024-02-21T23:46:34.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.195+0000 7f0edbfff700 1 -- 172.21.15.90:0/3188262850 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7f0eec003a00 con 0x7f0ef4100d80 2024-02-21T23:46:34.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.196+0000 7f0edbfff700 1 --2- 172.21.15.90:0/3188262850 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f0edc0a1120 0x7f0edc0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:34.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.196+0000 7f0ef2ffd700 1 --2- 172.21.15.90:0/3188262850 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f0edc0a1120 0x7f0edc0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:34.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.196+0000 7f0edbfff700 1 -- 172.21.15.90:0/3188262850 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7879+0+0 (secure 0 0 0) 0x7f0eec0cc270 con 0x7f0ef4100d80 2024-02-21T23:46:34.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.196+0000 7f0ef2ffd700 1 --2- 172.21.15.90:0/3188262850 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f0edc0a1120 0x7f0edc0a35e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4100240 tx=0x7f0ee400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:34.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.200+0000 7f0edbfff700 1 -- 172.21.15.90:0/3188262850 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f0eec095c40 con 0x7f0ef4100d80 2024-02-21T23:46:34.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:34 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/799478952' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-02-21T23:46:34.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:34 smithi176 ceph-mon[47172]: osdmap e78: 12 total, 12 up, 12 in 2024-02-21T23:46:34.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:34 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/4236800159' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-02-21T23:46:34.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:34 smithi176 ceph-mon[47172]: pgmap v218: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:34.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:34 smithi176 ceph-mon[47172]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:34.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:34 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/799478952' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-02-21T23:46:34.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:34 smithi178 ceph-mon[47547]: osdmap e78: 12 total, 12 up, 12 in 2024-02-21T23:46:34.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:34 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/4236800159' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-02-21T23:46:34.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:34 smithi178 ceph-mon[47547]: pgmap v218: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:34.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:34 smithi178 ceph-mon[47547]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:34.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.439+0000 7f0ef3fff700 1 -- 172.21.15.90:0/3188262850 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f0ef402cc70 con 0x7f0ef4100d80 2024-02-21T23:46:34.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.446+0000 7f0edbfff700 1 -- 172.21.15.90:0/3188262850 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f0eec0b7e60 con 0x7f0ef4100d80 2024-02-21T23:46:34.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.451+0000 7f0ef3fff700 1 -- 172.21.15.90:0/3188262850 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f0edc0a1120 msgr2=0x7f0edc0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:34.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.451+0000 7f0ef3fff700 1 --2- 172.21.15.90:0/3188262850 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f0edc0a1120 0x7f0edc0a35e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4100240 tx=0x7f0ee400b410 comp rx=0 tx=0).stop 2024-02-21T23:46:34.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.451+0000 7f0ef3fff700 1 -- 172.21.15.90:0/3188262850 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ef4100d80 msgr2=0x7f0ef419de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:34.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.451+0000 7f0ef3fff700 1 --2- 172.21.15.90:0/3188262850 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ef4100d80 0x7f0ef419de20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0eec00ef30 tx=0x7f0eec00c3a0 comp rx=0 tx=0).stop 2024-02-21T23:46:34.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.453+0000 7f0ef3fff700 1 -- 172.21.15.90:0/3188262850 shutdown_connections 2024-02-21T23:46:34.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.453+0000 7f0ef3fff700 1 --2- 172.21.15.90:0/3188262850 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f0edc0a1120 0x7f0edc0a35e0 secure :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4100240 tx=0x7f0ee400b410 comp rx=0 tx=0).stop 2024-02-21T23:46:34.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.453+0000 7f0ef3fff700 1 --2- 172.21.15.90:0/3188262850 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ef4100d80 0x7f0ef419de20 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:34.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.453+0000 7f0ef3fff700 1 --2- 172.21.15.90:0/3188262850 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0ef40ff1f0 0x7f0ef410c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:34.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.453+0000 7f0ef3fff700 1 --2- 172.21.15.90:0/3188262850 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0ef41003e0 0x7f0ef419b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:34.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.453+0000 7f0ef3fff700 1 -- 172.21.15.90:0/3188262850 >> 172.21.15.90:0/3188262850 conn(0x7f0ef40fa760 msgr2=0x7f0ef40fbb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:34.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.455+0000 7f0ef3fff700 1 -- 172.21.15.90:0/3188262850 shutdown_connections 2024-02-21T23:46:34.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.455+0000 7f0ef3fff700 1 -- 172.21.15.90:0/3188262850 wait complete. 2024-02-21T23:46:34.449 INFO:teuthology.orchestra.run.smithi090.stderr:new fs with metadata pool 2 and data pool 3 2024-02-21T23:46:34.464 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:34 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/799478952' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-02-21T23:46:34.464 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:34 smithi090 ceph-mon[43385]: osdmap e78: 12 total, 12 up, 12 in 2024-02-21T23:46:34.464 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:34 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/4236800159' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-02-21T23:46:34.464 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:34 smithi090 ceph-mon[43385]: pgmap v218: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:34.464 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:34 smithi090 ceph-mon[43385]: from='mgr.14150 172.21.15.90:0/3079828287' entity='mgr.x' 2024-02-21T23:46:34.465 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-02-21T23:46:34.465 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-02-21T23:46:34.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.607+0000 7f08f1abd700 1 -- 172.21.15.90:0/2105852508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08ec0ffbf0 msgr2=0x7f08ec100010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:34.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.607+0000 7f08f1abd700 1 --2- 172.21.15.90:0/2105852508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08ec0ffbf0 0x7f08ec100010 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f08dc0099c0 tx=0x7f08dc009cd0 comp rx=0 tx=0).stop 2024-02-21T23:46:34.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.607+0000 7f08f1abd700 1 -- 172.21.15.90:0/2105852508 shutdown_connections 2024-02-21T23:46:34.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.607+0000 7f08f1abd700 1 --2- 172.21.15.90:0/2105852508 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f08ec101780 0x7f08ec108030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:34.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.607+0000 7f08f1abd700 1 --2- 172.21.15.90:0/2105852508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08ec0ffbf0 0x7f08ec100010 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:34.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.607+0000 7f08f1abd700 1 --2- 172.21.15.90:0/2105852508 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f08ec100de0 0x7f08ec101240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:34.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.607+0000 7f08f1abd700 1 -- 172.21.15.90:0/2105852508 >> 172.21.15.90:0/2105852508 conn(0x7f08ec0fb160 msgr2=0x7f08ec0fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:34.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.607+0000 7f08f1abd700 1 -- 172.21.15.90:0/2105852508 shutdown_connections 2024-02-21T23:46:34.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.607+0000 7f08f1abd700 1 -- 172.21.15.90:0/2105852508 wait complete. 2024-02-21T23:46:34.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.608+0000 7f08f1abd700 1 Processor -- start 2024-02-21T23:46:34.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.608+0000 7f08f1abd700 1 -- start start 2024-02-21T23:46:34.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.608+0000 7f08f1abd700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f08ec0ffbf0 0x7f08ec069dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:34.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.608+0000 7f08f1abd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08ec100de0 0x7f08ec19ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:34.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.608+0000 7f08f1abd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f08ec101780 0x7f08ec19e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:34.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.608+0000 7f08f1abd700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec19e810 con 0x7f08ec100de0 2024-02-21T23:46:34.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.608+0000 7f08f1abd700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec19e980 con 0x7f08ec101780 2024-02-21T23:46:34.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.608+0000 7f08f1abd700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec19eb40 con 0x7f08ec0ffbf0 2024-02-21T23:46:34.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.608+0000 7f08eaffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08ec100de0 0x7f08ec19ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:34.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.608+0000 7f08eaffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08ec100de0 0x7f08ec19ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41656/0 (socket says 172.21.15.90:41656) 2024-02-21T23:46:34.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.608+0000 7f08eaffd700 1 -- 172.21.15.90:0/1625592747 learned_addr learned my addr 172.21.15.90:0/1625592747 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:34.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.608+0000 7f08eb7fe700 1 --2- 172.21.15.90:0/1625592747 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f08ec0ffbf0 0x7f08ec069dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:34.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.608+0000 7f08ebfff700 1 --2- 172.21.15.90:0/1625592747 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f08ec101780 0x7f08ec19e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:34.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.608+0000 7f08eaffd700 1 -- 172.21.15.90:0/1625592747 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f08ec0ffbf0 msgr2=0x7f08ec069dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:34.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.608+0000 7f08eaffd700 1 --2- 172.21.15.90:0/1625592747 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f08ec0ffbf0 0x7f08ec069dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:34.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.608+0000 7f08eaffd700 1 -- 172.21.15.90:0/1625592747 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f08ec101780 msgr2=0x7f08ec19e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:34.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.608+0000 7f08eaffd700 1 --2- 172.21.15.90:0/1625592747 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f08ec101780 0x7f08ec19e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:34.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.608+0000 7f08eaffd700 1 -- 172.21.15.90:0/1625592747 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08dc0095f0 con 0x7f08ec100de0 2024-02-21T23:46:34.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.609+0000 7f08eaffd700 1 --2- 172.21.15.90:0/1625592747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08ec100de0 0x7f08ec19ba00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f08e000d950 tx=0x7f08e000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:34.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.609+0000 7f08e8ff9700 1 -- 172.21.15.90:0/1625592747 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e0009b40 con 0x7f08ec100de0 2024-02-21T23:46:34.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.609+0000 7f08f1abd700 1 -- 172.21.15.90:0/1625592747 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08ec19edc0 con 0x7f08ec100de0 2024-02-21T23:46:34.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.609+0000 7f08e8ff9700 1 -- 172.21.15.90:0/1625592747 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f08e0009ca0 con 0x7f08ec100de0 2024-02-21T23:46:34.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.609+0000 7f08e8ff9700 1 -- 172.21.15.90:0/1625592747 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e000f620 con 0x7f08ec100de0 2024-02-21T23:46:34.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.609+0000 7f08f1abd700 1 -- 172.21.15.90:0/1625592747 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08ec19f270 con 0x7f08ec100de0 2024-02-21T23:46:34.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.610+0000 7f08f1abd700 1 -- 172.21.15.90:0/1625592747 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08ec107060 con 0x7f08ec100de0 2024-02-21T23:46:34.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.611+0000 7f08e8ff9700 1 -- 172.21.15.90:0/1625592747 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7f08e0010460 con 0x7f08ec100de0 2024-02-21T23:46:34.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.611+0000 7f08e8ff9700 1 --2- 172.21.15.90:0/1625592747 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f08d40a11a0 0x7f08d40a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:34.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.612+0000 7f08eb7fe700 1 --2- 172.21.15.90:0/1625592747 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f08d40a11a0 0x7f08d40a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:34.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.612+0000 7f08e8ff9700 1 -- 172.21.15.90:0/1625592747 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f08e00ccbd0 con 0x7f08ec100de0 2024-02-21T23:46:34.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.612+0000 7f08eb7fe700 1 --2- 172.21.15.90:0/1625592747 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f08d40a11a0 0x7f08d40a3660 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f08dc00b580 tx=0x7f08dc005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:34.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.615+0000 7f08e8ff9700 1 -- 172.21.15.90:0/1625592747 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f08e0096410 con 0x7f08ec100de0 2024-02-21T23:46:34.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:34.822+0000 7f08f1abd700 1 -- 172.21.15.90:0/1625592747 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f08ec067ed0 con 0x7f08ec100de0 2024-02-21T23:46:35.335 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:35 smithi176 ceph-mon[47172]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-21T23:46:35.335 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:35 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/4236800159' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-02-21T23:46:35.335 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:35 smithi176 ceph-mon[47172]: osdmap e79: 12 total, 12 up, 12 in 2024-02-21T23:46:35.335 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:35 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/3188262850' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-02-21T23:46:35.335 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:35 smithi176 ceph-mon[47172]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-02-21T23:46:35.335 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:35 smithi176 ceph-mon[47172]: osdmap e80: 12 total, 12 up, 12 in 2024-02-21T23:46:35.336 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:35 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/3188262850' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-02-21T23:46:35.336 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:35 smithi176 ceph-mon[47172]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-02-21T23:46:35.336 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:35 smithi176 ceph-mon[47172]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-02-21T23:46:35.336 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:35 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/1625592747' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-02-21T23:46:35.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:35 smithi178 ceph-mon[47547]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-21T23:46:35.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:35 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/4236800159' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-02-21T23:46:35.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:35 smithi178 ceph-mon[47547]: osdmap e79: 12 total, 12 up, 12 in 2024-02-21T23:46:35.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:35 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/3188262850' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-02-21T23:46:35.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:35 smithi178 ceph-mon[47547]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-02-21T23:46:35.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:35 smithi178 ceph-mon[47547]: osdmap e80: 12 total, 12 up, 12 in 2024-02-21T23:46:35.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:35 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/3188262850' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-02-21T23:46:35.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:35 smithi178 ceph-mon[47547]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-02-21T23:46:35.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:35 smithi178 ceph-mon[47547]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-02-21T23:46:35.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:35 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/1625592747' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-02-21T23:46:35.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.455+0000 7f08e8ff9700 1 -- 172.21.15.90:0/1625592747 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v81) v1 ==== 157+0+0 (secure 0 0 0) 0x7f08e00200e0 con 0x7f08ec100de0 2024-02-21T23:46:35.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.457+0000 7f08f1abd700 1 -- 172.21.15.90:0/1625592747 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f08d40a11a0 msgr2=0x7f08d40a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:35.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.457+0000 7f08f1abd700 1 --2- 172.21.15.90:0/1625592747 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f08d40a11a0 0x7f08d40a3660 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f08dc00b580 tx=0x7f08dc005fb0 comp rx=0 tx=0).stop 2024-02-21T23:46:35.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.457+0000 7f08f1abd700 1 -- 172.21.15.90:0/1625592747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08ec100de0 msgr2=0x7f08ec19ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:35.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.457+0000 7f08f1abd700 1 --2- 172.21.15.90:0/1625592747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08ec100de0 0x7f08ec19ba00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f08e000d950 tx=0x7f08e000dd10 comp rx=0 tx=0).stop 2024-02-21T23:46:35.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.457+0000 7f08f1abd700 1 -- 172.21.15.90:0/1625592747 shutdown_connections 2024-02-21T23:46:35.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.457+0000 7f08f1abd700 1 --2- 172.21.15.90:0/1625592747 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f08d40a11a0 0x7f08d40a3660 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:35.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.457+0000 7f08f1abd700 1 --2- 172.21.15.90:0/1625592747 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f08ec101780 0x7f08ec19e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:35.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.457+0000 7f08f1abd700 1 --2- 172.21.15.90:0/1625592747 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f08ec0ffbf0 0x7f08ec069dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:35.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.457+0000 7f08f1abd700 1 --2- 172.21.15.90:0/1625592747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08ec100de0 0x7f08ec19ba00 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:35.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.457+0000 7f08f1abd700 1 -- 172.21.15.90:0/1625592747 >> 172.21.15.90:0/1625592747 conn(0x7f08ec0fb160 msgr2=0x7f08ec1041a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:35.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.457+0000 7f08f1abd700 1 -- 172.21.15.90:0/1625592747 shutdown_connections 2024-02-21T23:46:35.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.457+0000 7f08f1abd700 1 -- 172.21.15.90:0/1625592747 wait complete. 2024-02-21T23:46:35.455 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:35 smithi090 ceph-mon[43385]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-21T23:46:35.455 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:35 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/4236800159' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-02-21T23:46:35.455 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:35 smithi090 ceph-mon[43385]: osdmap e79: 12 total, 12 up, 12 in 2024-02-21T23:46:35.455 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:35 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/3188262850' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-02-21T23:46:35.455 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:35 smithi090 ceph-mon[43385]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-02-21T23:46:35.455 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:35 smithi090 ceph-mon[43385]: osdmap e80: 12 total, 12 up, 12 in 2024-02-21T23:46:35.455 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:35 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/3188262850' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-02-21T23:46:35.455 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:35 smithi090 ceph-mon[43385]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-02-21T23:46:35.456 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:35 smithi090 ceph-mon[43385]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-02-21T23:46:35.456 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:35 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/1625592747' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-02-21T23:46:35.457 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-02-21T23:46:35.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.606+0000 7fae1efc6700 1 -- 172.21.15.90:0/1058826946 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae18100820 msgr2=0x7fae18100ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:35.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.606+0000 7fae1efc6700 1 --2- 172.21.15.90:0/1058826946 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae18100820 0x7fae18100ca0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fae1000b0a0 tx=0x7fae1000b3b0 comp rx=0 tx=0).stop 2024-02-21T23:46:35.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.606+0000 7fae1efc6700 1 -- 172.21.15.90:0/1058826946 shutdown_connections 2024-02-21T23:46:35.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.606+0000 7fae1efc6700 1 --2- 172.21.15.90:0/1058826946 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae18100820 0x7fae18100ca0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:35.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.606+0000 7fae1efc6700 1 --2- 172.21.15.90:0/1058826946 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae180ff500 0x7fae180ff920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:35.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.606+0000 7fae1efc6700 1 --2- 172.21.15.90:0/1058826946 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae180ffe60 0x7fae181002e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:35.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.606+0000 7fae1efc6700 1 -- 172.21.15.90:0/1058826946 >> 172.21.15.90:0/1058826946 conn(0x7fae180fb080 msgr2=0x7fae180fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:35.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.606+0000 7fae1efc6700 1 -- 172.21.15.90:0/1058826946 shutdown_connections 2024-02-21T23:46:35.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.606+0000 7fae1efc6700 1 -- 172.21.15.90:0/1058826946 wait complete. 2024-02-21T23:46:35.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.607+0000 7fae1efc6700 1 Processor -- start 2024-02-21T23:46:35.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.607+0000 7fae1efc6700 1 -- start start 2024-02-21T23:46:35.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.607+0000 7fae1efc6700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae180ff500 0x7fae18198eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:35.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.607+0000 7fae1efc6700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae180ffe60 0x7fae181993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:35.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.607+0000 7fae1efc6700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae18100820 0x7fae181a0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:35.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.607+0000 7fae1efc6700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae181a09e0 con 0x7fae180ff500 2024-02-21T23:46:35.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.607+0000 7fae1efc6700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae181a0b20 con 0x7fae180ffe60 2024-02-21T23:46:35.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.607+0000 7fae1efc6700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae181a0cb0 con 0x7fae18100820 2024-02-21T23:46:35.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.607+0000 7fae1cd62700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae180ff500 0x7fae18198eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:35.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.607+0000 7fae1cd62700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae180ff500 0x7fae18198eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41674/0 (socket says 172.21.15.90:41674) 2024-02-21T23:46:35.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.607+0000 7fae1cd62700 1 -- 172.21.15.90:0/889696222 learned_addr learned my addr 172.21.15.90:0/889696222 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:35.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.607+0000 7fae1d563700 1 --2- 172.21.15.90:0/889696222 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae18100820 0x7fae181a0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:35.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.607+0000 7fae17fff700 1 --2- 172.21.15.90:0/889696222 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae180ffe60 0x7fae181993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:35.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.607+0000 7fae1cd62700 1 -- 172.21.15.90:0/889696222 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae18100820 msgr2=0x7fae181a0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:35.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.607+0000 7fae1cd62700 1 --2- 172.21.15.90:0/889696222 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae18100820 0x7fae181a0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:35.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.607+0000 7fae1cd62700 1 -- 172.21.15.90:0/889696222 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae180ffe60 msgr2=0x7fae181993f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:35.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.607+0000 7fae1cd62700 1 --2- 172.21.15.90:0/889696222 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae180ffe60 0x7fae181993f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:35.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.607+0000 7fae1cd62700 1 -- 172.21.15.90:0/889696222 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae10009d00 con 0x7fae180ff500 2024-02-21T23:46:35.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.607+0000 7fae17fff700 1 --2- 172.21.15.90:0/889696222 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae180ffe60 0x7fae181993f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:46:35.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.607+0000 7fae1cd62700 1 --2- 172.21.15.90:0/889696222 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae180ff500 0x7fae18198eb0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fae0c00b530 tx=0x7fae0c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:35.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.607+0000 7fae15ffb700 1 -- 172.21.15.90:0/889696222 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae0c011830 con 0x7fae180ff500 2024-02-21T23:46:35.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.608+0000 7fae15ffb700 1 -- 172.21.15.90:0/889696222 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fae0c011e50 con 0x7fae180ff500 2024-02-21T23:46:35.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.608+0000 7fae1efc6700 1 -- 172.21.15.90:0/889696222 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae181a0f90 con 0x7fae180ff500 2024-02-21T23:46:35.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.608+0000 7fae15ffb700 1 -- 172.21.15.90:0/889696222 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae0c00f2e0 con 0x7fae180ff500 2024-02-21T23:46:35.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.608+0000 7fae1efc6700 1 -- 172.21.15.90:0/889696222 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae181a13b0 con 0x7fae180ff500 2024-02-21T23:46:35.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.609+0000 7fae1efc6700 1 -- 172.21.15.90:0/889696222 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae1810a4e0 con 0x7fae180ff500 2024-02-21T23:46:35.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.610+0000 7fae15ffb700 1 -- 172.21.15.90:0/889696222 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7fae0c010380 con 0x7fae180ff500 2024-02-21T23:46:35.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.610+0000 7fae15ffb700 1 --2- 172.21.15.90:0/889696222 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fae080a1110 0x7fae080a35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:35.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.610+0000 7fae17fff700 1 --2- 172.21.15.90:0/889696222 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fae080a1110 0x7fae080a35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:35.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.610+0000 7fae15ffb700 1 -- 172.21.15.90:0/889696222 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8205+0+0 (secure 0 0 0) 0x7fae0c0cc410 con 0x7fae180ff500 2024-02-21T23:46:35.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.611+0000 7fae17fff700 1 --2- 172.21.15.90:0/889696222 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fae080a1110 0x7fae080a35d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fae04005950 tx=0x7fae040058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:35.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.615+0000 7fae15ffb700 1 -- 172.21.15.90:0/889696222 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fae0c095bc0 con 0x7fae180ff500 2024-02-21T23:46:35.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:35.812+0000 7fae1efc6700 1 -- 172.21.15.90:0/889696222 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fae180679a0 con 0x7fae180ff500 2024-02-21T23:46:36.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:36 smithi090 ceph-mon[43385]: pgmap v221: 97 pgs: 4 creating+peering, 92 unknown, 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:36.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:36 smithi090 ceph-mon[43385]: mds.? [v2:172.21.15.176:6838/875627757,v1:172.21.15.176:6839/875627757] up:active 2024-02-21T23:46:36.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:36 smithi090 ceph-mon[43385]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-02-21T23:46:36.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:36 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/1625592747' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-02-21T23:46:36.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:36 smithi090 ceph-mon[43385]: osdmap e81: 12 total, 12 up, 12 in 2024-02-21T23:46:36.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:36 smithi090 ceph-mon[43385]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-02-21T23:46:36.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:36 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/889696222' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-02-21T23:46:36.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:36 smithi090 ceph-mon[43385]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-21T23:46:36.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:36 smithi090 ceph-mon[43385]: Cluster is now healthy 2024-02-21T23:46:36.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:36 smithi176 ceph-mon[47172]: pgmap v221: 97 pgs: 4 creating+peering, 92 unknown, 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:36.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:36 smithi176 ceph-mon[47172]: mds.? [v2:172.21.15.176:6838/875627757,v1:172.21.15.176:6839/875627757] up:active 2024-02-21T23:46:36.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:36 smithi176 ceph-mon[47172]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-02-21T23:46:36.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:36 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/1625592747' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-02-21T23:46:36.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:36 smithi176 ceph-mon[47172]: osdmap e81: 12 total, 12 up, 12 in 2024-02-21T23:46:36.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:36 smithi176 ceph-mon[47172]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-02-21T23:46:36.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:36 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/889696222' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-02-21T23:46:36.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:36 smithi176 ceph-mon[47172]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-21T23:46:36.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:36 smithi176 ceph-mon[47172]: Cluster is now healthy 2024-02-21T23:46:36.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:36 smithi178 ceph-mon[47547]: pgmap v221: 97 pgs: 4 creating+peering, 92 unknown, 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-21T23:46:36.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:36 smithi178 ceph-mon[47547]: mds.? [v2:172.21.15.176:6838/875627757,v1:172.21.15.176:6839/875627757] up:active 2024-02-21T23:46:36.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:36 smithi178 ceph-mon[47547]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-02-21T23:46:36.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:36 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/1625592747' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-02-21T23:46:36.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:36 smithi178 ceph-mon[47547]: osdmap e81: 12 total, 12 up, 12 in 2024-02-21T23:46:36.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:36 smithi178 ceph-mon[47547]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-02-21T23:46:36.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:36 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/889696222' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-02-21T23:46:36.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:36 smithi178 ceph-mon[47547]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-21T23:46:36.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:36 smithi178 ceph-mon[47547]: Cluster is now healthy 2024-02-21T23:46:37.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.457+0000 7fae15ffb700 1 -- 172.21.15.90:0/889696222 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v83) v1 ==== 228+0+0 (secure 0 0 0) 0x7fae0c0991e0 con 0x7fae180ff500 2024-02-21T23:46:37.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.461+0000 7fae1efc6700 1 -- 172.21.15.90:0/889696222 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fae080a1110 msgr2=0x7fae080a35d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:37.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.461+0000 7fae1efc6700 1 --2- 172.21.15.90:0/889696222 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fae080a1110 0x7fae080a35d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fae04005950 tx=0x7fae040058e0 comp rx=0 tx=0).stop 2024-02-21T23:46:37.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.461+0000 7fae1efc6700 1 -- 172.21.15.90:0/889696222 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae180ff500 msgr2=0x7fae18198eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:37.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.461+0000 7fae1efc6700 1 --2- 172.21.15.90:0/889696222 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae180ff500 0x7fae18198eb0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fae0c00b530 tx=0x7fae0c00b840 comp rx=0 tx=0).stop 2024-02-21T23:46:37.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.462+0000 7fae1efc6700 1 -- 172.21.15.90:0/889696222 shutdown_connections 2024-02-21T23:46:37.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.462+0000 7fae1efc6700 1 --2- 172.21.15.90:0/889696222 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fae080a1110 0x7fae080a35d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:37.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.462+0000 7fae1efc6700 1 --2- 172.21.15.90:0/889696222 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae18100820 0x7fae181a0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:37.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.462+0000 7fae1efc6700 1 --2- 172.21.15.90:0/889696222 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae180ff500 0x7fae18198eb0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:37.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.462+0000 7fae1efc6700 1 --2- 172.21.15.90:0/889696222 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae180ffe60 0x7fae181993f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:37.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.462+0000 7fae1efc6700 1 -- 172.21.15.90:0/889696222 >> 172.21.15.90:0/889696222 conn(0x7fae180fb080 msgr2=0x7fae180fc800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:37.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.462+0000 7fae1efc6700 1 -- 172.21.15.90:0/889696222 shutdown_connections 2024-02-21T23:46:37.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.462+0000 7fae1efc6700 1 -- 172.21.15.90:0/889696222 wait complete. 2024-02-21T23:46:37.450 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_data_ec' created 2024-02-21T23:46:37.462 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-02-21T23:46:37.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.603+0000 7feadc500700 1 -- 172.21.15.90:0/4143684712 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead41006e0 msgr2=0x7fead4100f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:37.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.603+0000 7feadc500700 1 --2- 172.21.15.90:0/4143684712 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead41006e0 0x7fead4100f60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7feacc00b0a0 tx=0x7feacc00b3b0 comp rx=0 tx=0).stop 2024-02-21T23:46:37.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.603+0000 7feadc500700 1 -- 172.21.15.90:0/4143684712 shutdown_connections 2024-02-21T23:46:37.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.603+0000 7feadc500700 1 --2- 172.21.15.90:0/4143684712 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead41006e0 0x7fead4100f60 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:37.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.603+0000 7feadc500700 1 --2- 172.21.15.90:0/4143684712 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fead40fefc0 0x7fead40ff3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:37.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.603+0000 7feadc500700 1 --2- 172.21.15.90:0/4143684712 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fead40ffd20 0x7fead41001a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:37.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.603+0000 7feadc500700 1 -- 172.21.15.90:0/4143684712 >> 172.21.15.90:0/4143684712 conn(0x7fead40fa8f0 msgr2=0x7fead40fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:37.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.603+0000 7feadc500700 1 -- 172.21.15.90:0/4143684712 shutdown_connections 2024-02-21T23:46:37.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.603+0000 7feadc500700 1 -- 172.21.15.90:0/4143684712 wait complete. 2024-02-21T23:46:37.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.604+0000 7feadc500700 1 Processor -- start 2024-02-21T23:46:37.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.604+0000 7feadc500700 1 -- start start 2024-02-21T23:46:37.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.604+0000 7feadc500700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fead40fefc0 0x7fead4198f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:37.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.604+0000 7feadc500700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fead40ffd20 0x7fead4199440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:37.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.604+0000 7feadc500700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead41006e0 0x7fead41a0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:37.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.604+0000 7feadc500700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead41a0a30 con 0x7fead41006e0 2024-02-21T23:46:37.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.604+0000 7feadc500700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead41a0b70 con 0x7fead40fefc0 2024-02-21T23:46:37.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.604+0000 7feadc500700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead41a0d00 con 0x7fead40ffd20 2024-02-21T23:46:37.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.604+0000 7feadaa9d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead41006e0 0x7fead41a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:37.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.604+0000 7feadaa9d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead41006e0 0x7fead41a0460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41690/0 (socket says 172.21.15.90:41690) 2024-02-21T23:46:37.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.604+0000 7feadaa9d700 1 -- 172.21.15.90:0/1708770591 learned_addr learned my addr 172.21.15.90:0/1708770591 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:37.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.604+0000 7fead9a9b700 1 --2- 172.21.15.90:0/1708770591 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fead40ffd20 0x7fead4199440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:37.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.604+0000 7feada29c700 1 --2- 172.21.15.90:0/1708770591 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fead40fefc0 0x7fead4198f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:37.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.604+0000 7feadaa9d700 1 -- 172.21.15.90:0/1708770591 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fead40ffd20 msgr2=0x7fead4199440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:37.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.604+0000 7feadaa9d700 1 --2- 172.21.15.90:0/1708770591 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fead40ffd20 0x7fead4199440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:37.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.604+0000 7feadaa9d700 1 -- 172.21.15.90:0/1708770591 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fead40fefc0 msgr2=0x7fead4198f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:37.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.604+0000 7feadaa9d700 1 --2- 172.21.15.90:0/1708770591 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fead40fefc0 0x7fead4198f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:37.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.604+0000 7feadaa9d700 1 -- 172.21.15.90:0/1708770591 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feacc009d00 con 0x7fead41006e0 2024-02-21T23:46:37.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.604+0000 7feadaa9d700 1 --2- 172.21.15.90:0/1708770591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead41006e0 0x7fead41a0460 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7feacc007ab0 tx=0x7feacc004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:37.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.605+0000 7feacb7fe700 1 -- 172.21.15.90:0/1708770591 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feacc00e040 con 0x7fead41006e0 2024-02-21T23:46:37.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.605+0000 7feacb7fe700 1 -- 172.21.15.90:0/1708770591 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feacc004e70 con 0x7fead41006e0 2024-02-21T23:46:37.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.605+0000 7feadc500700 1 -- 172.21.15.90:0/1708770591 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fead41a0f80 con 0x7fead41006e0 2024-02-21T23:46:37.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.605+0000 7feacb7fe700 1 -- 172.21.15.90:0/1708770591 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feacc01da70 con 0x7fead41006e0 2024-02-21T23:46:37.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.605+0000 7feadc500700 1 -- 172.21.15.90:0/1708770591 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fead41a1340 con 0x7fead41006e0 2024-02-21T23:46:37.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.606+0000 7feadc500700 1 -- 172.21.15.90:0/1708770591 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fead410ac60 con 0x7fead41006e0 2024-02-21T23:46:37.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.607+0000 7feacb7fe700 1 -- 172.21.15.90:0/1708770591 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7feacc0058a0 con 0x7fead41006e0 2024-02-21T23:46:37.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.608+0000 7feacb7fe700 1 --2- 172.21.15.90:0/1708770591 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7feac00a11a0 0x7feac00a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:37.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.608+0000 7feada29c700 1 --2- 172.21.15.90:0/1708770591 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7feac00a11a0 0x7feac00a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:37.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.608+0000 7feacb7fe700 1 -- 172.21.15.90:0/1708770591 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8698+0+0 (secure 0 0 0) 0x7feacc0ce340 con 0x7fead41006e0 2024-02-21T23:46:37.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.608+0000 7feada29c700 1 --2- 172.21.15.90:0/1708770591 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7feac00a11a0 0x7feac00a3660 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7feac4005950 tx=0x7feac400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:37.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.612+0000 7feacb7fe700 1 -- 172.21.15.90:0/1708770591 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7feacc097990 con 0x7fead41006e0 2024-02-21T23:46:37.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:37 smithi090 ceph-mon[43385]: osdmap e82: 12 total, 12 up, 12 in 2024-02-21T23:46:37.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:37.809+0000 7feadc500700 1 -- 172.21.15.90:0/1708770591 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7fead40671f0 con 0x7fead41006e0 2024-02-21T23:46:37.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:37 smithi176 ceph-mon[47172]: osdmap e82: 12 total, 12 up, 12 in 2024-02-21T23:46:37.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:37 smithi178 ceph-mon[47547]: osdmap e82: 12 total, 12 up, 12 in 2024-02-21T23:46:38.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.461+0000 7feacb7fe700 1 -- 172.21.15.90:0/1708770591 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v84) v1 ==== 169+0+0 (secure 0 0 0) 0x7feacc09afb0 con 0x7fead41006e0 2024-02-21T23:46:38.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.469+0000 7feadc500700 1 -- 172.21.15.90:0/1708770591 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7feac00a11a0 msgr2=0x7feac00a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:38.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.469+0000 7feadc500700 1 --2- 172.21.15.90:0/1708770591 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7feac00a11a0 0x7feac00a3660 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7feac4005950 tx=0x7feac400b410 comp rx=0 tx=0).stop 2024-02-21T23:46:38.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.469+0000 7feadc500700 1 -- 172.21.15.90:0/1708770591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead41006e0 msgr2=0x7fead41a0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:38.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.469+0000 7feadc500700 1 --2- 172.21.15.90:0/1708770591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead41006e0 0x7fead41a0460 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7feacc007ab0 tx=0x7feacc004c80 comp rx=0 tx=0).stop 2024-02-21T23:46:38.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.470+0000 7feadc500700 1 -- 172.21.15.90:0/1708770591 shutdown_connections 2024-02-21T23:46:38.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.470+0000 7feadc500700 1 --2- 172.21.15.90:0/1708770591 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7feac00a11a0 0x7feac00a3660 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:38.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.470+0000 7feadc500700 1 --2- 172.21.15.90:0/1708770591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead41006e0 0x7fead41a0460 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:38.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.470+0000 7feadc500700 1 --2- 172.21.15.90:0/1708770591 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fead40fefc0 0x7fead4198f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:38.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.470+0000 7feadc500700 1 --2- 172.21.15.90:0/1708770591 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fead40ffd20 0x7fead4199440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:38.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.470+0000 7feadc500700 1 -- 172.21.15.90:0/1708770591 >> 172.21.15.90:0/1708770591 conn(0x7fead40fa8f0 msgr2=0x7fead40fbf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:38.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.477+0000 7feadc500700 1 -- 172.21.15.90:0/1708770591 shutdown_connections 2024-02-21T23:46:38.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.479+0000 7feadc500700 1 -- 172.21.15.90:0/1708770591 wait complete. 2024-02-21T23:46:38.468 INFO:teuthology.orchestra.run.smithi090.stderr:set pool 4 allow_ec_overwrites to true 2024-02-21T23:46:38.479 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-02-21T23:46:38.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.619+0000 7fa692471700 1 -- 172.21.15.90:0/1511610749 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa68c0ff6f0 msgr2=0x7fa68c0ffba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:38.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.619+0000 7fa692471700 1 --2- 172.21.15.90:0/1511610749 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa68c0ff6f0 0x7fa68c0ffba0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa68400b0a0 tx=0x7fa68400b3b0 comp rx=0 tx=0).stop 2024-02-21T23:46:38.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.619+0000 7fa692471700 1 -- 172.21.15.90:0/1511610749 shutdown_connections 2024-02-21T23:46:38.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.619+0000 7fa692471700 1 --2- 172.21.15.90:0/1511610749 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa68c0ff6f0 0x7fa68c0ffba0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:38.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.619+0000 7fa692471700 1 --2- 172.21.15.90:0/1511610749 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa68c1011d0 0x7fa68c1015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:38.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.619+0000 7fa692471700 1 --2- 172.21.15.90:0/1511610749 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa68c0fed50 0x7fa68c0ff1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:38.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.619+0000 7fa692471700 1 -- 172.21.15.90:0/1511610749 >> 172.21.15.90:0/1511610749 conn(0x7fa68c0fa8f0 msgr2=0x7fa68c0fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:38.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.619+0000 7fa692471700 1 -- 172.21.15.90:0/1511610749 shutdown_connections 2024-02-21T23:46:38.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.619+0000 7fa692471700 1 -- 172.21.15.90:0/1511610749 wait complete. 2024-02-21T23:46:38.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.620+0000 7fa692471700 1 Processor -- start 2024-02-21T23:46:38.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.620+0000 7fa692471700 1 -- start start 2024-02-21T23:46:38.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.620+0000 7fa692471700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa68c0fed50 0x7fa68c198f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:38.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.620+0000 7fa692471700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa68c0ff6f0 0x7fa68c1994a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:38.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.620+0000 7fa692471700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa68c1011d0 0x7fa68c1a04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:38.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.620+0000 7fa692471700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa68c1a0a90 con 0x7fa68c1011d0 2024-02-21T23:46:38.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.620+0000 7fa692471700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa68c1a0bd0 con 0x7fa68c0fed50 2024-02-21T23:46:38.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.620+0000 7fa692471700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa68c1a0d60 con 0x7fa68c0ff6f0 2024-02-21T23:46:38.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.621+0000 7fa690a0e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa68c1011d0 0x7fa68c1a04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:38.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.621+0000 7fa690a0e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa68c1011d0 0x7fa68c1a04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49152/0 (socket says 172.21.15.90:49152) 2024-02-21T23:46:38.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.621+0000 7fa690a0e700 1 -- 172.21.15.90:0/448290034 learned_addr learned my addr 172.21.15.90:0/448290034 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:38.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.621+0000 7fa68bfff700 1 --2- 172.21.15.90:0/448290034 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa68c0fed50 0x7fa68c198f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:38.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.621+0000 7fa6837fe700 1 --2- 172.21.15.90:0/448290034 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa68c0ff6f0 0x7fa68c1994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:38.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.621+0000 7fa690a0e700 1 -- 172.21.15.90:0/448290034 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa68c0ff6f0 msgr2=0x7fa68c1994a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:38.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.621+0000 7fa690a0e700 1 --2- 172.21.15.90:0/448290034 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa68c0ff6f0 0x7fa68c1994a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:38.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.621+0000 7fa690a0e700 1 -- 172.21.15.90:0/448290034 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa68c0fed50 msgr2=0x7fa68c198f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:38.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.621+0000 7fa690a0e700 1 --2- 172.21.15.90:0/448290034 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa68c0fed50 0x7fa68c198f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:38.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.621+0000 7fa690a0e700 1 -- 172.21.15.90:0/448290034 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa684009d00 con 0x7fa68c1011d0 2024-02-21T23:46:38.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.621+0000 7fa68bfff700 1 --2- 172.21.15.90:0/448290034 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa68c0fed50 0x7fa68c198f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:46:38.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.621+0000 7fa690a0e700 1 --2- 172.21.15.90:0/448290034 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa68c1011d0 0x7fa68c1a04c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa684009030 tx=0x7fa68400bb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:38.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.621+0000 7fa689ffb700 1 -- 172.21.15.90:0/448290034 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa68400e040 con 0x7fa68c1011d0 2024-02-21T23:46:38.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.621+0000 7fa689ffb700 1 -- 172.21.15.90:0/448290034 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa684004e60 con 0x7fa68c1011d0 2024-02-21T23:46:38.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.621+0000 7fa692471700 1 -- 172.21.15.90:0/448290034 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa68c1a0fe0 con 0x7fa68c1011d0 2024-02-21T23:46:38.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.621+0000 7fa689ffb700 1 -- 172.21.15.90:0/448290034 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa684005370 con 0x7fa68c1011d0 2024-02-21T23:46:38.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.621+0000 7fa692471700 1 -- 172.21.15.90:0/448290034 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa68c1a1400 con 0x7fa68c1011d0 2024-02-21T23:46:38.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.622+0000 7fa692471700 1 -- 172.21.15.90:0/448290034 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa68c10ae30 con 0x7fa68c1011d0 2024-02-21T23:46:38.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.623+0000 7fa689ffb700 1 -- 172.21.15.90:0/448290034 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7fa68401d9c0 con 0x7fa68c1011d0 2024-02-21T23:46:38.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.624+0000 7fa689ffb700 1 --2- 172.21.15.90:0/448290034 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fa6780a1120 0x7fa6780a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:38.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.624+0000 7fa689ffb700 1 -- 172.21.15.90:0/448290034 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8698+0+0 (secure 0 0 0) 0x7fa6840ce4a0 con 0x7fa68c1011d0 2024-02-21T23:46:38.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.624+0000 7fa68bfff700 1 --2- 172.21.15.90:0/448290034 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fa6780a1120 0x7fa6780a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:38.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.624+0000 7fa68bfff700 1 --2- 172.21.15.90:0/448290034 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fa6780a1120 0x7fa6780a35e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa67c00bc10 tx=0x7fa67c00b5f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:38.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.627+0000 7fa689ffb700 1 -- 172.21.15.90:0/448290034 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fa684097af0 con 0x7fa68c1011d0 2024-02-21T23:46:38.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:38 smithi090 ceph-mon[43385]: pgmap v224: 97 pgs: 4 creating+peering, 24 unknown, 69 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2024-02-21T23:46:38.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:38 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/889696222' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-02-21T23:46:38.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:38 smithi090 ceph-mon[43385]: osdmap e83: 12 total, 12 up, 12 in 2024-02-21T23:46:38.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:38 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/1708770591' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-02-21T23:46:38.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:38 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/1708770591' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-02-21T23:46:38.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:38 smithi090 ceph-mon[43385]: osdmap e84: 12 total, 12 up, 12 in 2024-02-21T23:46:38.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:38 smithi176 ceph-mon[47172]: pgmap v224: 97 pgs: 4 creating+peering, 24 unknown, 69 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2024-02-21T23:46:38.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:38 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/889696222' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-02-21T23:46:38.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:38 smithi176 ceph-mon[47172]: osdmap e83: 12 total, 12 up, 12 in 2024-02-21T23:46:38.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:38 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/1708770591' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-02-21T23:46:38.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:38 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/1708770591' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-02-21T23:46:38.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:38 smithi176 ceph-mon[47172]: osdmap e84: 12 total, 12 up, 12 in 2024-02-21T23:46:38.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.877+0000 7fa692471700 1 -- 172.21.15.90:0/448290034 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7fa68c02d090 con 0x7fa68c1011d0 2024-02-21T23:46:38.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.882+0000 7fa689ffb700 1 -- 172.21.15.90:0/448290034 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) v1 ==== 137+0+0 (secure 0 0 0) 0x7fa68409b110 con 0x7fa68c1011d0 2024-02-21T23:46:38.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.884+0000 7fa692471700 1 -- 172.21.15.90:0/448290034 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fa6780a1120 msgr2=0x7fa6780a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:38.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.884+0000 7fa692471700 1 --2- 172.21.15.90:0/448290034 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fa6780a1120 0x7fa6780a35e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa67c00bc10 tx=0x7fa67c00b5f0 comp rx=0 tx=0).stop 2024-02-21T23:46:38.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.884+0000 7fa692471700 1 -- 172.21.15.90:0/448290034 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa68c1011d0 msgr2=0x7fa68c1a04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:38.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.884+0000 7fa692471700 1 --2- 172.21.15.90:0/448290034 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa68c1011d0 0x7fa68c1a04c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa684009030 tx=0x7fa68400bb60 comp rx=0 tx=0).stop 2024-02-21T23:46:38.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.884+0000 7fa692471700 1 -- 172.21.15.90:0/448290034 shutdown_connections 2024-02-21T23:46:38.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.884+0000 7fa692471700 1 --2- 172.21.15.90:0/448290034 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fa6780a1120 0x7fa6780a35e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:38.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.884+0000 7fa692471700 1 --2- 172.21.15.90:0/448290034 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa68c1011d0 0x7fa68c1a04c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:38.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.884+0000 7fa692471700 1 --2- 172.21.15.90:0/448290034 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa68c0fed50 0x7fa68c198f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:38.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.884+0000 7fa692471700 1 --2- 172.21.15.90:0/448290034 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa68c0ff6f0 0x7fa68c1994a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:38.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.885+0000 7fa692471700 1 -- 172.21.15.90:0/448290034 >> 172.21.15.90:0/448290034 conn(0x7fa68c0fa8f0 msgr2=0x7fa68c0fc040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:38.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.885+0000 7fa692471700 1 -- 172.21.15.90:0/448290034 shutdown_connections 2024-02-21T23:46:38.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:38.885+0000 7fa692471700 1 -- 172.21.15.90:0/448290034 wait complete. 2024-02-21T23:46:38.871 INFO:teuthology.orchestra.run.smithi090.stderr:added data pool 4 to fsmap 2024-02-21T23:46:38.881 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:46:38.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:38 smithi178 ceph-mon[47547]: pgmap v224: 97 pgs: 4 creating+peering, 24 unknown, 69 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2024-02-21T23:46:38.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:38 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/889696222' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-02-21T23:46:38.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:38 smithi178 ceph-mon[47547]: osdmap e83: 12 total, 12 up, 12 in 2024-02-21T23:46:38.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:38 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/1708770591' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-02-21T23:46:38.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:38 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/1708770591' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-02-21T23:46:38.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:38 smithi178 ceph-mon[47547]: osdmap e84: 12 total, 12 up, 12 in 2024-02-21T23:46:39.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.022+0000 7f037c94e700 1 -- 172.21.15.90:0/833418132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0378100d10 msgr2=0x7f0378101170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:39.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.022+0000 7f037c94e700 1 --2- 172.21.15.90:0/833418132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0378100d10 0x7f0378101170 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f036c0099c0 tx=0x7f036c009cd0 comp rx=0 tx=0).stop 2024-02-21T23:46:39.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.022+0000 7f037c94e700 1 -- 172.21.15.90:0/833418132 shutdown_connections 2024-02-21T23:46:39.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.022+0000 7f037c94e700 1 --2- 172.21.15.90:0/833418132 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f03781016b0 0x7f0378107f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:39.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.022+0000 7f037c94e700 1 --2- 172.21.15.90:0/833418132 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f03780ffb20 0x7f03780fff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:39.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.022+0000 7f037c94e700 1 --2- 172.21.15.90:0/833418132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0378100d10 0x7f0378101170 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:39.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.022+0000 7f037c94e700 1 -- 172.21.15.90:0/833418132 >> 172.21.15.90:0/833418132 conn(0x7f03780fb090 msgr2=0x7f03780fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:39.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.022+0000 7f037c94e700 1 -- 172.21.15.90:0/833418132 shutdown_connections 2024-02-21T23:46:39.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.022+0000 7f037c94e700 1 -- 172.21.15.90:0/833418132 wait complete. 2024-02-21T23:46:39.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.022+0000 7f037c94e700 1 Processor -- start 2024-02-21T23:46:39.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.023+0000 7f037c94e700 1 -- start start 2024-02-21T23:46:39.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.023+0000 7f037c94e700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f03780ffb20 0x7f0378069ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:39.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.023+0000 7f037c94e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0378100d10 0x7f037819b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:39.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.023+0000 7f037c94e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f03781016b0 0x7f037819de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:39.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.023+0000 7f037c94e700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f037819e680 con 0x7f0378100d10 2024-02-21T23:46:39.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.023+0000 7f037c94e700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f037819e7f0 con 0x7f03781016b0 2024-02-21T23:46:39.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.023+0000 7f037c94e700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f037819e9b0 con 0x7f03780ffb20 2024-02-21T23:46:39.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.023+0000 7f0375d9b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0378100d10 0x7f037819b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:39.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.023+0000 7f0375d9b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0378100d10 0x7f037819b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49170/0 (socket says 172.21.15.90:49170) 2024-02-21T23:46:39.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.023+0000 7f0375d9b700 1 -- 172.21.15.90:0/805340993 learned_addr learned my addr 172.21.15.90:0/805340993 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:39.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.023+0000 7f037659c700 1 --2- 172.21.15.90:0/805340993 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f03780ffb20 0x7f0378069ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:39.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.023+0000 7f0376d9d700 1 --2- 172.21.15.90:0/805340993 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f03781016b0 0x7f037819de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:39.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.023+0000 7f0375d9b700 1 -- 172.21.15.90:0/805340993 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f03780ffb20 msgr2=0x7f0378069ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:39.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.023+0000 7f0375d9b700 1 --2- 172.21.15.90:0/805340993 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f03780ffb20 0x7f0378069ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:39.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.023+0000 7f0375d9b700 1 -- 172.21.15.90:0/805340993 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f03781016b0 msgr2=0x7f037819de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:39.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.023+0000 7f0375d9b700 1 --2- 172.21.15.90:0/805340993 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f03781016b0 0x7f037819de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:39.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.023+0000 7f0375d9b700 1 -- 172.21.15.90:0/805340993 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f036c0095f0 con 0x7f0378100d10 2024-02-21T23:46:39.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.023+0000 7f0375d9b700 1 --2- 172.21.15.90:0/805340993 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0378100d10 0x7f037819b870 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f036c003730 tx=0x7f036c00f670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:39.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.023+0000 7f03677fe700 1 -- 172.21.15.90:0/805340993 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036c01d070 con 0x7f0378100d10 2024-02-21T23:46:39.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.023+0000 7f037c94e700 1 -- 172.21.15.90:0/805340993 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f037819ebd0 con 0x7f0378100d10 2024-02-21T23:46:39.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.023+0000 7f03677fe700 1 -- 172.21.15.90:0/805340993 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f036c0059a0 con 0x7f0378100d10 2024-02-21T23:46:39.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.024+0000 7f03677fe700 1 -- 172.21.15.90:0/805340993 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036c004a00 con 0x7f0378100d10 2024-02-21T23:46:39.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.024+0000 7f037c94e700 1 -- 172.21.15.90:0/805340993 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f037819f080 con 0x7f0378100d10 2024-02-21T23:46:39.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.025+0000 7f03657fa700 1 -- 172.21.15.90:0/805340993 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f035c0052f0 con 0x7f0378100d10 2024-02-21T23:46:39.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.026+0000 7f03677fe700 1 -- 172.21.15.90:0/805340993 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7f036c005210 con 0x7f0378100d10 2024-02-21T23:46:39.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.026+0000 7f03677fe700 1 --2- 172.21.15.90:0/805340993 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f03600a1120 0x7f03600a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:39.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.026+0000 7f03677fe700 1 -- 172.21.15.90:0/805340993 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f036c0cda40 con 0x7f0378100d10 2024-02-21T23:46:39.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.026+0000 7f037659c700 1 --2- 172.21.15.90:0/805340993 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f03600a1120 0x7f03600a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:39.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.027+0000 7f037659c700 1 --2- 172.21.15.90:0/805340993 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f03600a1120 0x7f03600a35e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0378100b70 tx=0x7f0368009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:39.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.030+0000 7f03677fe700 1 -- 172.21.15.90:0/805340993 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f036c097070 con 0x7f0378100d10 2024-02-21T23:46:39.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.271+0000 7f03657fa700 1 -- 172.21.15.90:0/805340993 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f035c005160 con 0x7f0378100d10 2024-02-21T23:46:39.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.272+0000 7f03677fe700 1 -- 172.21.15.90:0/805340993 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+10816 (secure 0 0 0) 0x7f036c09a690 con 0x7f0378100d10 2024-02-21T23:46:39.257 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:46:39.257 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/1591404300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1591404300},{"type":"v1","addr":"172.21.15.90:6835","nonce":1591404300}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/1441666263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":1441666263},{"type":"v1","addr":"172.21.15.90:6837","nonce":1441666263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14607,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6839/732535159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":732535159},{"type":"v1","addr":"172.21.15.90:6839","nonce":732535159}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14619,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/3572584154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":3572584154},{"type":"v1","addr":"172.21.15.90:6841","nonce":3572584154}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1403694104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1403694104},{"type":"v1","addr":"172.21.15.178:6833","nonce":1403694104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/2956287083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":2956287083},{"type":"v1","addr":"172.21.15.176:6833","nonce":2956287083}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/3430010314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":3430010314},{"type":"v1","addr":"172.21.15.176:6835","nonce":3430010314}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/1260308729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1260308729},{"type":"v1","addr":"172.21.15.178:6835","nonce":1260308729}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/4166803229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":4166803229},{"type":"v1","addr":"172.21.15.178:6837","nonce":4166803229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24458,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/4127802836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":4127802836},{"type":"v1","addr":"172.21.15.176:6837","nonce":4127802836}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24470,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6839/3726236229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":3726236229},{"type":"v1","addr":"172.21.15.178:6839","nonce":3726236229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:46:34.442043+0000","modified":"2024-02-21T23:46:38.878999+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.176:6839/875627757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6838","nonce":875627757},{"type":"v1","addr":"172.21.15.176:6839","nonce":875627757}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-02-21T23:46:39.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.274+0000 7f03657fa700 1 -- 172.21.15.90:0/805340993 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f03600a1120 msgr2=0x7f03600a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:39.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.274+0000 7f03657fa700 1 --2- 172.21.15.90:0/805340993 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f03600a1120 0x7f03600a35e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0378100b70 tx=0x7f0368009260 comp rx=0 tx=0).stop 2024-02-21T23:46:39.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.274+0000 7f03657fa700 1 -- 172.21.15.90:0/805340993 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0378100d10 msgr2=0x7f037819b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:39.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.274+0000 7f03657fa700 1 --2- 172.21.15.90:0/805340993 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0378100d10 0x7f037819b870 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f036c003730 tx=0x7f036c00f670 comp rx=0 tx=0).stop 2024-02-21T23:46:39.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.274+0000 7f03657fa700 1 -- 172.21.15.90:0/805340993 shutdown_connections 2024-02-21T23:46:39.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.274+0000 7f03657fa700 1 --2- 172.21.15.90:0/805340993 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f03600a1120 0x7f03600a35e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:39.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.274+0000 7f03657fa700 1 --2- 172.21.15.90:0/805340993 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f03781016b0 0x7f037819de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:39.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.274+0000 7f03657fa700 1 --2- 172.21.15.90:0/805340993 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f03780ffb20 0x7f0378069ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:39.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.274+0000 7f03657fa700 1 --2- 172.21.15.90:0/805340993 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0378100d10 0x7f037819b870 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:39.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.274+0000 7f03657fa700 1 -- 172.21.15.90:0/805340993 >> 172.21.15.90:0/805340993 conn(0x7f03780fb090 msgr2=0x7f03781040e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:39.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.274+0000 7f03657fa700 1 -- 172.21.15.90:0/805340993 shutdown_connections 2024-02-21T23:46:39.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.274+0000 7f03657fa700 1 -- 172.21.15.90:0/805340993 wait complete. 2024-02-21T23:46:39.260 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 17 2024-02-21T23:46:39.271 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:46:39.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.408+0000 7f843d362700 1 -- 172.21.15.90:0/1176312871 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8438101680 msgr2=0x7f8438107f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:39.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.408+0000 7f843d362700 1 --2- 172.21.15.90:0/1176312871 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8438101680 0x7f8438107f30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f843000b0d0 tx=0x7f843000b3e0 comp rx=0 tx=0).stop 2024-02-21T23:46:39.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.409+0000 7f843d362700 1 -- 172.21.15.90:0/1176312871 shutdown_connections 2024-02-21T23:46:39.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.409+0000 7f843d362700 1 --2- 172.21.15.90:0/1176312871 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8438101680 0x7f8438107f30 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:39.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.409+0000 7f843d362700 1 --2- 172.21.15.90:0/1176312871 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84380ffaf0 0x7f84380fff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:39.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.409+0000 7f843d362700 1 --2- 172.21.15.90:0/1176312871 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8438100ce0 0x7f8438101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:39.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.409+0000 7f843d362700 1 -- 172.21.15.90:0/1176312871 >> 172.21.15.90:0/1176312871 conn(0x7f84380fb060 msgr2=0x7f84380fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:39.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.409+0000 7f843d362700 1 -- 172.21.15.90:0/1176312871 shutdown_connections 2024-02-21T23:46:39.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.409+0000 7f843d362700 1 -- 172.21.15.90:0/1176312871 wait complete. 2024-02-21T23:46:39.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.409+0000 7f843d362700 1 Processor -- start 2024-02-21T23:46:39.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.409+0000 7f843d362700 1 -- start start 2024-02-21T23:46:39.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.409+0000 7f843d362700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84380ffaf0 0x7f84380688c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:39.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.409+0000 7f843d362700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8438100ce0 0x7f843819b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:39.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.409+0000 7f843d362700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8438101680 0x7f843819df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:39.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.410+0000 7f843d362700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f843819e780 con 0x7f8438101680 2024-02-21T23:46:39.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.410+0000 7f843d362700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f843819e8f0 con 0x7f84380ffaf0 2024-02-21T23:46:39.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.410+0000 7f843d362700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f843819eab0 con 0x7f8438100ce0 2024-02-21T23:46:39.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.410+0000 7f84377fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8438101680 0x7f843819df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:39.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.410+0000 7f84377fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8438101680 0x7f843819df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49194/0 (socket says 172.21.15.90:49194) 2024-02-21T23:46:39.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.410+0000 7f84377fe700 1 -- 172.21.15.90:0/725991941 learned_addr learned my addr 172.21.15.90:0/725991941 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:39.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.410+0000 7f84367fc700 1 --2- 172.21.15.90:0/725991941 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8438100ce0 0x7f843819b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:39.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.410+0000 7f8436ffd700 1 --2- 172.21.15.90:0/725991941 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84380ffaf0 0x7f84380688c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:39.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.410+0000 7f84377fe700 1 -- 172.21.15.90:0/725991941 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8438100ce0 msgr2=0x7f843819b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:39.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.410+0000 7f84377fe700 1 --2- 172.21.15.90:0/725991941 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8438100ce0 0x7f843819b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:39.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.410+0000 7f84377fe700 1 -- 172.21.15.90:0/725991941 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84380ffaf0 msgr2=0x7f84380688c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:39.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.410+0000 7f84377fe700 1 --2- 172.21.15.90:0/725991941 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84380ffaf0 0x7f84380688c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:39.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.410+0000 7f84377fe700 1 -- 172.21.15.90:0/725991941 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8430009d00 con 0x7f8438101680 2024-02-21T23:46:39.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.410+0000 7f84377fe700 1 --2- 172.21.15.90:0/725991941 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8438101680 0x7f843819df90 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8430007bd0 tx=0x7f8430009440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:39.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.410+0000 7f842ffff700 1 -- 172.21.15.90:0/725991941 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843000e040 con 0x7f8438101680 2024-02-21T23:46:39.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.410+0000 7f842ffff700 1 -- 172.21.15.90:0/725991941 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f843001dd50 con 0x7f8438101680 2024-02-21T23:46:39.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.410+0000 7f843d362700 1 -- 172.21.15.90:0/725991941 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f843819ecd0 con 0x7f8438101680 2024-02-21T23:46:39.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.410+0000 7f842ffff700 1 -- 172.21.15.90:0/725991941 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843001b450 con 0x7f8438101680 2024-02-21T23:46:39.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.410+0000 7f843d362700 1 -- 172.21.15.90:0/725991941 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f843819f150 con 0x7f8438101680 2024-02-21T23:46:39.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.411+0000 7f843d362700 1 -- 172.21.15.90:0/725991941 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8438106ea0 con 0x7f8438101680 2024-02-21T23:46:39.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.413+0000 7f842ffff700 1 -- 172.21.15.90:0/725991941 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7f843001d890 con 0x7f8438101680 2024-02-21T23:46:39.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.413+0000 7f842ffff700 1 --2- 172.21.15.90:0/725991941 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f84240a1120 0x7f84240a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:39.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.413+0000 7f8436ffd700 1 --2- 172.21.15.90:0/725991941 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f84240a1120 0x7f84240a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:39.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.413+0000 7f842ffff700 1 -- 172.21.15.90:0/725991941 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f84300cf130 con 0x7f8438101680 2024-02-21T23:46:39.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.414+0000 7f8436ffd700 1 --2- 172.21.15.90:0/725991941 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f84240a1120 0x7f84240a35e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8428005950 tx=0x7f84280058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:39.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.417+0000 7f842ffff700 1 -- 172.21.15.90:0/725991941 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f8430098760 con 0x7f8438101680 2024-02-21T23:46:39.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.620+0000 7f843d362700 1 -- 172.21.15.90:0/725991941 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f843802d100 con 0x7f8438101680 2024-02-21T23:46:39.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.621+0000 7f842ffff700 1 -- 172.21.15.90:0/725991941 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23925 (secure 0 0 0) 0x7f84300a2070 con 0x7f8438101680 2024-02-21T23:46:39.606 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:46:39.606 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":85,"fsid":"7ed81218-d112-11ee-95bd-87774f69a715","created":"2024-02-21T23:40:41.785526+0000","modified":"2024-02-21T23:46:38.878698+0000","last_up_change":"2024-02-21T23:46:00.860501+0000","last_in_change":"2024-02-21T23:45:44.423815+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:43:21.675789+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:46:32.337047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-21T23:46:33.622440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-21T23:46:36.686123+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"77878890-d9a5-4d5b-a943-fec29ee0446e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6803","nonce":885319826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6805","nonce":885319826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6809","nonce":885319826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6807","nonce":885319826}]},"public_addr":"172.21.15.90:6803/885319826","cluster_addr":"172.21.15.90:6805/885319826","heartbeat_back_addr":"172.21.15.90:6809/885319826","heartbeat_front_addr":"172.21.15.90:6807/885319826","state":["exists","up"]},{"osd":1,"uuid":"7d6ea950-7a57-4f13-ab38-10a658912783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6801","nonce":4085240925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6803","nonce":4085240925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6807","nonce":4085240925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6805","nonce":4085240925}]},"public_addr":"172.21.15.176:6801/4085240925","cluster_addr":"172.21.15.176:6803/4085240925","heartbeat_back_addr":"172.21.15.176:6807/4085240925","heartbeat_front_addr":"172.21.15.176:6805/4085240925","state":["exists","up"]},{"osd":2,"uuid":"b83486bb-27f2-4d56-97fb-fb336fec4b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6801","nonce":3597250789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6803","nonce":3597250789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6807","nonce":3597250789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6805","nonce":3597250789}]},"public_addr":"172.21.15.178:6801/3597250789","cluster_addr":"172.21.15.178:6803/3597250789","heartbeat_back_addr":"172.21.15.178:6807/3597250789","heartbeat_front_addr":"172.21.15.178:6805/3597250789","state":["exists","up"]},{"osd":3,"uuid":"870e5bb6-eed5-4a0a-8e03-38de4f816b44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6811","nonce":3299043821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6813","nonce":3299043821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6817","nonce":3299043821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6815","nonce":3299043821}]},"public_addr":"172.21.15.90:6811/3299043821","cluster_addr":"172.21.15.90:6813/3299043821","heartbeat_back_addr":"172.21.15.90:6817/3299043821","heartbeat_front_addr":"172.21.15.90:6815/3299043821","state":["exists","up"]},{"osd":4,"uuid":"cb9f7f48-76e0-4368-ab87-2a09a8487ccd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6809","nonce":3548915687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6811","nonce":3548915687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6815","nonce":3548915687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6813","nonce":3548915687}]},"public_addr":"172.21.15.176:6809/3548915687","cluster_addr":"172.21.15.176:6811/3548915687","heartbeat_back_addr":"172.21.15.176:6815/3548915687","heartbeat_front_addr":"172.21.15.176:6813/3548915687","state":["exists","up"]},{"osd":5,"uuid":"54749022-c50d-4831-a1d6-8412b59c8a01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6809","nonce":286274518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6811","nonce":286274518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6815","nonce":286274518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6813","nonce":286274518}]},"public_addr":"172.21.15.178:6809/286274518","cluster_addr":"172.21.15.178:6811/286274518","heartbeat_back_addr":"172.21.15.178:6815/286274518","heartbeat_front_addr":"172.21.15.178:6813/286274518","state":["exists","up"]},{"osd":6,"uuid":"09bb5393-2efd-4a63-b0e8-99bef5bced13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6819","nonce":67640620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6821","nonce":67640620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6825","nonce":67640620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6823","nonce":67640620}]},"public_addr":"172.21.15.90:6819/67640620","cluster_addr":"172.21.15.90:6821/67640620","heartbeat_back_addr":"172.21.15.90:6825/67640620","heartbeat_front_addr":"172.21.15.90:6823/67640620","state":["exists","up"]},{"osd":7,"uuid":"7a859e69-6b63-467b-a3b3-99247d66f5c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6817","nonce":3998561598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6819","nonce":3998561598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6823","nonce":3998561598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6821","nonce":3998561598}]},"public_addr":"172.21.15.176:6817/3998561598","cluster_addr":"172.21.15.176:6819/3998561598","heartbeat_back_addr":"172.21.15.176:6823/3998561598","heartbeat_front_addr":"172.21.15.176:6821/3998561598","state":["exists","up"]},{"osd":8,"uuid":"bc23144e-22a5-4894-a8b5-9323383574d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6817","nonce":490229963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6819","nonce":490229963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6823","nonce":490229963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6821","nonce":490229963}]},"public_addr":"172.21.15.178:6817/490229963","cluster_addr":"172.21.15.178:6819/490229963","heartbeat_back_addr":"172.21.15.178:6823/490229963","heartbeat_front_addr":"172.21.15.178:6821/490229963","state":["exists","up"]},{"osd":9,"uuid":"0c2cecd4-55de-4fc4-a6f4-ca97c8460444","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6827","nonce":2061394761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6829","nonce":2061394761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6833","nonce":2061394761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6831","nonce":2061394761}]},"public_addr":"172.21.15.90:6827/2061394761","cluster_addr":"172.21.15.90:6829/2061394761","heartbeat_back_addr":"172.21.15.90:6833/2061394761","heartbeat_front_addr":"172.21.15.90:6831/2061394761","state":["exists","up"]},{"osd":10,"uuid":"fa04232b-b1ec-447a-8e94-7c9bf26d0291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6825","nonce":639411757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6827","nonce":639411757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6831","nonce":639411757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6829","nonce":639411757}]},"public_addr":"172.21.15.176:6825/639411757","cluster_addr":"172.21.15.176:6827/639411757","heartbeat_back_addr":"172.21.15.176:6831/639411757","heartbeat_front_addr":"172.21.15.176:6829/639411757","state":["exists","up"]},{"osd":11,"uuid":"7282e294-64fc-4856-9c74-86115f69408c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6825","nonce":4144536256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6827","nonce":4144536256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6831","nonce":4144536256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6829","nonce":4144536256}]},"public_addr":"172.21.15.178:6825/4144536256","cluster_addr":"172.21.15.178:6827/4144536256","heartbeat_back_addr":"172.21.15.178:6831/4144536256","heartbeat_front_addr":"172.21.15.178:6829/4144536256","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:42:44.280345+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:01.235630+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:19.408896+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:36.201749+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:53.401854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:44:11.342158+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:44:29.324502+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:44:46.821595+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:04.642344+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:23.019004+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:41.140149+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:59.401951+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/159655312":"2024-02-22T23:41:33.360021+0000","172.21.15.90:6801/1385551604":"2024-02-22T23:41:33.360021+0000","172.21.15.90:0/1452574770":"2024-02-22T23:41:33.360021+0000","172.21.15.90:0/3815292343":"2024-02-22T23:41:11.484145+0000","172.21.15.90:6800/1385551604":"2024-02-22T23:41:33.360021+0000","172.21.15.90:6801/4247695574":"2024-02-22T23:41:11.484145+0000","172.21.15.90:0/1328266622":"2024-02-22T23:41:11.484145+0000","172.21.15.90:6800/4247695574":"2024-02-22T23:41:11.484145+0000","172.21.15.90:0/2569421522":"2024-02-22T23:41:11.484145+0000","172.21.15.90:0/3019701027":"2024-02-22T23:41:33.360021+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:46:39.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.623+0000 7f843d362700 1 -- 172.21.15.90:0/725991941 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f84240a1120 msgr2=0x7f84240a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:39.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.623+0000 7f843d362700 1 --2- 172.21.15.90:0/725991941 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f84240a1120 0x7f84240a35e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8428005950 tx=0x7f84280058e0 comp rx=0 tx=0).stop 2024-02-21T23:46:39.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.623+0000 7f843d362700 1 -- 172.21.15.90:0/725991941 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8438101680 msgr2=0x7f843819df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:39.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.623+0000 7f843d362700 1 --2- 172.21.15.90:0/725991941 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8438101680 0x7f843819df90 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8430007bd0 tx=0x7f8430009440 comp rx=0 tx=0).stop 2024-02-21T23:46:39.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.623+0000 7f843d362700 1 -- 172.21.15.90:0/725991941 shutdown_connections 2024-02-21T23:46:39.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.623+0000 7f843d362700 1 --2- 172.21.15.90:0/725991941 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f84240a1120 0x7f84240a35e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:39.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.623+0000 7f843d362700 1 --2- 172.21.15.90:0/725991941 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8438101680 0x7f843819df90 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:39.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.623+0000 7f843d362700 1 --2- 172.21.15.90:0/725991941 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84380ffaf0 0x7f84380688c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:39.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.623+0000 7f843d362700 1 --2- 172.21.15.90:0/725991941 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8438100ce0 0x7f843819b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:39.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.623+0000 7f843d362700 1 -- 172.21.15.90:0/725991941 >> 172.21.15.90:0/725991941 conn(0x7f84380fb060 msgr2=0x7f8438103fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:39.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.623+0000 7f843d362700 1 -- 172.21.15.90:0/725991941 shutdown_connections 2024-02-21T23:46:39.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.623+0000 7f843d362700 1 -- 172.21.15.90:0/725991941 wait complete. 2024-02-21T23:46:39.619 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:39 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/448290034' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-02-21T23:46:39.619 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:39 smithi090 ceph-mon[43385]: osdmap e85: 12 total, 12 up, 12 in 2024-02-21T23:46:39.619 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:39 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/448290034' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-02-21T23:46:39.619 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:39 smithi090 ceph-mon[43385]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-02-21T23:46:39.619 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:39 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/805340993' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-21T23:46:39.621 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:46:39.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.758+0000 7f18ad75c700 1 -- 172.21.15.90:0/1074446787 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18a8102790 msgr2=0x7f18a8109040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:39.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.758+0000 7f18ad75c700 1 --2- 172.21.15.90:0/1074446787 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18a8102790 0x7f18a8109040 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f18a000b0d0 tx=0x7f18a000b3e0 comp rx=0 tx=0).stop 2024-02-21T23:46:39.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.758+0000 7f18ad75c700 1 -- 172.21.15.90:0/1074446787 shutdown_connections 2024-02-21T23:46:39.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.758+0000 7f18ad75c700 1 --2- 172.21.15.90:0/1074446787 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18a8102790 0x7f18a8109040 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:39.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.758+0000 7f18ad75c700 1 --2- 172.21.15.90:0/1074446787 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18a8100c00 0x7f18a8101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:39.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.758+0000 7f18ad75c700 1 --2- 172.21.15.90:0/1074446787 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f18a8101df0 0x7f18a8102250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:39.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.758+0000 7f18ad75c700 1 -- 172.21.15.90:0/1074446787 >> 172.21.15.90:0/1074446787 conn(0x7f18a80fc170 msgr2=0x7f18a80fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:39.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.758+0000 7f18ad75c700 1 -- 172.21.15.90:0/1074446787 shutdown_connections 2024-02-21T23:46:39.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.758+0000 7f18ad75c700 1 -- 172.21.15.90:0/1074446787 wait complete. 2024-02-21T23:46:39.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.759+0000 7f18ad75c700 1 Processor -- start 2024-02-21T23:46:39.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.759+0000 7f18ad75c700 1 -- start start 2024-02-21T23:46:39.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.759+0000 7f18ad75c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18a8100c00 0x7f18a8196c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:39.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.759+0000 7f18ad75c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18a8101df0 0x7f18a8197170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:39.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.759+0000 7f18ad75c700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f18a8102790 0x7f18a819e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:39.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.759+0000 7f18ad75c700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a819e760 con 0x7f18a8101df0 2024-02-21T23:46:39.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.759+0000 7f18ad75c700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a819e8a0 con 0x7f18a8100c00 2024-02-21T23:46:39.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.759+0000 7f18ad75c700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a819ea30 con 0x7f18a8102790 2024-02-21T23:46:39.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.759+0000 7f18a67fc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18a8101df0 0x7f18a8197170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:39.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.759+0000 7f18a67fc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18a8101df0 0x7f18a8197170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49204/0 (socket says 172.21.15.90:49204) 2024-02-21T23:46:39.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.759+0000 7f18a67fc700 1 -- 172.21.15.90:0/202032530 learned_addr learned my addr 172.21.15.90:0/202032530 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:39.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.759+0000 7f18a77fe700 1 --2- 172.21.15.90:0/202032530 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f18a8102790 0x7f18a819e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:39.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.759+0000 7f18a6ffd700 1 --2- 172.21.15.90:0/202032530 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18a8100c00 0x7f18a8196c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:39.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.759+0000 7f18a67fc700 1 -- 172.21.15.90:0/202032530 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f18a8102790 msgr2=0x7f18a819e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:39.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.759+0000 7f18a67fc700 1 --2- 172.21.15.90:0/202032530 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f18a8102790 0x7f18a819e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:39.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.759+0000 7f18a67fc700 1 -- 172.21.15.90:0/202032530 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18a8100c00 msgr2=0x7f18a8196c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:39.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.759+0000 7f18a67fc700 1 --2- 172.21.15.90:0/202032530 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18a8100c00 0x7f18a8196c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:39.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.759+0000 7f18a67fc700 1 -- 172.21.15.90:0/202032530 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a0009d00 con 0x7f18a8101df0 2024-02-21T23:46:39.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.760+0000 7f18a67fc700 1 --2- 172.21.15.90:0/202032530 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18a8101df0 0x7f18a8197170 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f189c00e9a0 tx=0x7f189c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:39.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.760+0000 7f188ffff700 1 -- 172.21.15.90:0/202032530 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f189c009790 con 0x7f18a8101df0 2024-02-21T23:46:39.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.760+0000 7f18ad75c700 1 -- 172.21.15.90:0/202032530 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18a819ed10 con 0x7f18a8101df0 2024-02-21T23:46:39.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.760+0000 7f188ffff700 1 -- 172.21.15.90:0/202032530 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f189c010430 con 0x7f18a8101df0 2024-02-21T23:46:39.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.760+0000 7f188ffff700 1 -- 172.21.15.90:0/202032530 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f189c018680 con 0x7f18a8101df0 2024-02-21T23:46:39.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.760+0000 7f18ad75c700 1 -- 172.21.15.90:0/202032530 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18a819f1d0 con 0x7f18a8101df0 2024-02-21T23:46:39.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.761+0000 7f18ad75c700 1 -- 172.21.15.90:0/202032530 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18a8108030 con 0x7f18a8101df0 2024-02-21T23:46:39.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.762+0000 7f188ffff700 1 -- 172.21.15.90:0/202032530 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7f189c0105a0 con 0x7f18a8101df0 2024-02-21T23:46:39.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.762+0000 7f188ffff700 1 --2- 172.21.15.90:0/202032530 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f18900a1120 0x7f18900a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:39.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.762+0000 7f188ffff700 1 -- 172.21.15.90:0/202032530 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f189c014070 con 0x7f18a8101df0 2024-02-21T23:46:39.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.762+0000 7f18a6ffd700 1 --2- 172.21.15.90:0/202032530 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f18900a1120 0x7f18900a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:39.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.763+0000 7f18a6ffd700 1 --2- 172.21.15.90:0/202032530 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f18900a1120 0x7f18900a35e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f18a8101c50 tx=0x7f189800a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:39.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.766+0000 7f188ffff700 1 -- 172.21.15.90:0/202032530 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f189c096460 con 0x7f18a8101df0 2024-02-21T23:46:39.834 INFO:journalctl@ceph.mds.b.smithi176.stdout:Feb 21 23:46:39 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-b[83617]: 2024-02-21T23:46:39.380+0000 7fa8f0d25700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:46:39.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:39 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/448290034' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-02-21T23:46:39.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:39 smithi176 ceph-mon[47172]: osdmap e85: 12 total, 12 up, 12 in 2024-02-21T23:46:39.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:39 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/448290034' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-02-21T23:46:39.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:39 smithi176 ceph-mon[47172]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-02-21T23:46:39.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:39 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/805340993' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-21T23:46:39.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:39 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/448290034' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-02-21T23:46:39.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:39 smithi178 ceph-mon[47547]: osdmap e85: 12 total, 12 up, 12 in 2024-02-21T23:46:39.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:39 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/448290034' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-02-21T23:46:39.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:39 smithi178 ceph-mon[47547]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-02-21T23:46:39.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:39 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/805340993' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-21T23:46:39.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.969+0000 7f18ad75c700 1 -- 172.21.15.90:0/202032530 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18a802ce00 con 0x7f18a8101df0 2024-02-21T23:46:39.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.971+0000 7f188ffff700 1 -- 172.21.15.90:0/202032530 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23925 (secure 0 0 0) 0x7f189c026090 con 0x7f18a8101df0 2024-02-21T23:46:39.956 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:46:39.956 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":85,"fsid":"7ed81218-d112-11ee-95bd-87774f69a715","created":"2024-02-21T23:40:41.785526+0000","modified":"2024-02-21T23:46:38.878698+0000","last_up_change":"2024-02-21T23:46:00.860501+0000","last_in_change":"2024-02-21T23:45:44.423815+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:43:21.675789+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:46:32.337047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-21T23:46:33.622440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-21T23:46:36.686123+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"77878890-d9a5-4d5b-a943-fec29ee0446e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6803","nonce":885319826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6805","nonce":885319826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6809","nonce":885319826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6807","nonce":885319826}]},"public_addr":"172.21.15.90:6803/885319826","cluster_addr":"172.21.15.90:6805/885319826","heartbeat_back_addr":"172.21.15.90:6809/885319826","heartbeat_front_addr":"172.21.15.90:6807/885319826","state":["exists","up"]},{"osd":1,"uuid":"7d6ea950-7a57-4f13-ab38-10a658912783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6801","nonce":4085240925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6803","nonce":4085240925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6807","nonce":4085240925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6805","nonce":4085240925}]},"public_addr":"172.21.15.176:6801/4085240925","cluster_addr":"172.21.15.176:6803/4085240925","heartbeat_back_addr":"172.21.15.176:6807/4085240925","heartbeat_front_addr":"172.21.15.176:6805/4085240925","state":["exists","up"]},{"osd":2,"uuid":"b83486bb-27f2-4d56-97fb-fb336fec4b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6801","nonce":3597250789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6803","nonce":3597250789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6807","nonce":3597250789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6805","nonce":3597250789}]},"public_addr":"172.21.15.178:6801/3597250789","cluster_addr":"172.21.15.178:6803/3597250789","heartbeat_back_addr":"172.21.15.178:6807/3597250789","heartbeat_front_addr":"172.21.15.178:6805/3597250789","state":["exists","up"]},{"osd":3,"uuid":"870e5bb6-eed5-4a0a-8e03-38de4f816b44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6811","nonce":3299043821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6813","nonce":3299043821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6817","nonce":3299043821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6815","nonce":3299043821}]},"public_addr":"172.21.15.90:6811/3299043821","cluster_addr":"172.21.15.90:6813/3299043821","heartbeat_back_addr":"172.21.15.90:6817/3299043821","heartbeat_front_addr":"172.21.15.90:6815/3299043821","state":["exists","up"]},{"osd":4,"uuid":"cb9f7f48-76e0-4368-ab87-2a09a8487ccd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6809","nonce":3548915687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6811","nonce":3548915687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6815","nonce":3548915687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6813","nonce":3548915687}]},"public_addr":"172.21.15.176:6809/3548915687","cluster_addr":"172.21.15.176:6811/3548915687","heartbeat_back_addr":"172.21.15.176:6815/3548915687","heartbeat_front_addr":"172.21.15.176:6813/3548915687","state":["exists","up"]},{"osd":5,"uuid":"54749022-c50d-4831-a1d6-8412b59c8a01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6809","nonce":286274518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6811","nonce":286274518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6815","nonce":286274518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6813","nonce":286274518}]},"public_addr":"172.21.15.178:6809/286274518","cluster_addr":"172.21.15.178:6811/286274518","heartbeat_back_addr":"172.21.15.178:6815/286274518","heartbeat_front_addr":"172.21.15.178:6813/286274518","state":["exists","up"]},{"osd":6,"uuid":"09bb5393-2efd-4a63-b0e8-99bef5bced13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6819","nonce":67640620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6821","nonce":67640620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6825","nonce":67640620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6823","nonce":67640620}]},"public_addr":"172.21.15.90:6819/67640620","cluster_addr":"172.21.15.90:6821/67640620","heartbeat_back_addr":"172.21.15.90:6825/67640620","heartbeat_front_addr":"172.21.15.90:6823/67640620","state":["exists","up"]},{"osd":7,"uuid":"7a859e69-6b63-467b-a3b3-99247d66f5c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6817","nonce":3998561598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6819","nonce":3998561598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6823","nonce":3998561598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6821","nonce":3998561598}]},"public_addr":"172.21.15.176:6817/3998561598","cluster_addr":"172.21.15.176:6819/3998561598","heartbeat_back_addr":"172.21.15.176:6823/3998561598","heartbeat_front_addr":"172.21.15.176:6821/3998561598","state":["exists","up"]},{"osd":8,"uuid":"bc23144e-22a5-4894-a8b5-9323383574d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6817","nonce":490229963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6819","nonce":490229963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6823","nonce":490229963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6821","nonce":490229963}]},"public_addr":"172.21.15.178:6817/490229963","cluster_addr":"172.21.15.178:6819/490229963","heartbeat_back_addr":"172.21.15.178:6823/490229963","heartbeat_front_addr":"172.21.15.178:6821/490229963","state":["exists","up"]},{"osd":9,"uuid":"0c2cecd4-55de-4fc4-a6f4-ca97c8460444","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6827","nonce":2061394761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6829","nonce":2061394761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6833","nonce":2061394761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6831","nonce":2061394761}]},"public_addr":"172.21.15.90:6827/2061394761","cluster_addr":"172.21.15.90:6829/2061394761","heartbeat_back_addr":"172.21.15.90:6833/2061394761","heartbeat_front_addr":"172.21.15.90:6831/2061394761","state":["exists","up"]},{"osd":10,"uuid":"fa04232b-b1ec-447a-8e94-7c9bf26d0291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6825","nonce":639411757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6827","nonce":639411757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6831","nonce":639411757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6829","nonce":639411757}]},"public_addr":"172.21.15.176:6825/639411757","cluster_addr":"172.21.15.176:6827/639411757","heartbeat_back_addr":"172.21.15.176:6831/639411757","heartbeat_front_addr":"172.21.15.176:6829/639411757","state":["exists","up"]},{"osd":11,"uuid":"7282e294-64fc-4856-9c74-86115f69408c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6825","nonce":4144536256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6827","nonce":4144536256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6831","nonce":4144536256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6829","nonce":4144536256}]},"public_addr":"172.21.15.178:6825/4144536256","cluster_addr":"172.21.15.178:6827/4144536256","heartbeat_back_addr":"172.21.15.178:6831/4144536256","heartbeat_front_addr":"172.21.15.178:6829/4144536256","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:42:44.280345+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:01.235630+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:19.408896+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:36.201749+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:53.401854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:44:11.342158+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:44:29.324502+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:44:46.821595+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:04.642344+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:23.019004+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:41.140149+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:59.401951+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/159655312":"2024-02-22T23:41:33.360021+0000","172.21.15.90:6801/1385551604":"2024-02-22T23:41:33.360021+0000","172.21.15.90:0/1452574770":"2024-02-22T23:41:33.360021+0000","172.21.15.90:0/3815292343":"2024-02-22T23:41:11.484145+0000","172.21.15.90:6800/1385551604":"2024-02-22T23:41:33.360021+0000","172.21.15.90:6801/4247695574":"2024-02-22T23:41:11.484145+0000","172.21.15.90:0/1328266622":"2024-02-22T23:41:11.484145+0000","172.21.15.90:6800/4247695574":"2024-02-22T23:41:11.484145+0000","172.21.15.90:0/2569421522":"2024-02-22T23:41:11.484145+0000","172.21.15.90:0/3019701027":"2024-02-22T23:41:33.360021+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:46:39.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.973+0000 7f18ad75c700 1 -- 172.21.15.90:0/202032530 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f18900a1120 msgr2=0x7f18900a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:39.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.973+0000 7f18ad75c700 1 --2- 172.21.15.90:0/202032530 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f18900a1120 0x7f18900a35e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f18a8101c50 tx=0x7f189800a380 comp rx=0 tx=0).stop 2024-02-21T23:46:39.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.973+0000 7f18ad75c700 1 -- 172.21.15.90:0/202032530 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18a8101df0 msgr2=0x7f18a8197170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:39.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.973+0000 7f18ad75c700 1 --2- 172.21.15.90:0/202032530 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18a8101df0 0x7f18a8197170 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f189c00e9a0 tx=0x7f189c00ecb0 comp rx=0 tx=0).stop 2024-02-21T23:46:39.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.973+0000 7f18ad75c700 1 -- 172.21.15.90:0/202032530 shutdown_connections 2024-02-21T23:46:39.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.973+0000 7f18ad75c700 1 --2- 172.21.15.90:0/202032530 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f18900a1120 0x7f18900a35e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:39.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.973+0000 7f18ad75c700 1 --2- 172.21.15.90:0/202032530 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f18a8102790 0x7f18a819e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:39.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.973+0000 7f18ad75c700 1 --2- 172.21.15.90:0/202032530 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18a8100c00 0x7f18a8196c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:39.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.973+0000 7f18ad75c700 1 --2- 172.21.15.90:0/202032530 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18a8101df0 0x7f18a8197170 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:39.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.973+0000 7f18ad75c700 1 -- 172.21.15.90:0/202032530 >> 172.21.15.90:0/202032530 conn(0x7f18a80fc170 msgr2=0x7f18a8105170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:39.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.973+0000 7f18ad75c700 1 -- 172.21.15.90:0/202032530 shutdown_connections 2024-02-21T23:46:39.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:39.973+0000 7f18ad75c700 1 -- 172.21.15.90:0/202032530 wait complete. 2024-02-21T23:46:39.984 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:46:40.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.121+0000 7f02964ce700 1 -- 172.21.15.90:0/2955049849 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0290101100 msgr2=0x7f0290101520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:40.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.121+0000 7f02964ce700 1 --2- 172.21.15.90:0/2955049849 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0290101100 0x7f0290101520 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0280009990 tx=0x7f0280009ca0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.121+0000 7f02964ce700 1 -- 172.21.15.90:0/2955049849 shutdown_connections 2024-02-21T23:46:40.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.121+0000 7f02964ce700 1 --2- 172.21.15.90:0/2955049849 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0290100710 0x7f0290107fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.121+0000 7f02964ce700 1 --2- 172.21.15.90:0/2955049849 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0290101100 0x7f0290101520 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.121+0000 7f02964ce700 1 --2- 172.21.15.90:0/2955049849 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f02900ffd50 0x7f02901001d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.121+0000 7f02964ce700 1 -- 172.21.15.90:0/2955049849 >> 172.21.15.90:0/2955049849 conn(0x7f02900fa7e0 msgr2=0x7f02900fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:40.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.121+0000 7f02964ce700 1 -- 172.21.15.90:0/2955049849 shutdown_connections 2024-02-21T23:46:40.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.121+0000 7f02964ce700 1 -- 172.21.15.90:0/2955049849 wait complete. 2024-02-21T23:46:40.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.122+0000 7f02964ce700 1 Processor -- start 2024-02-21T23:46:40.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.122+0000 7f02964ce700 1 -- start start 2024-02-21T23:46:40.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.122+0000 7f02964ce700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02900ffd50 0x7f02900ff760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:40.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.122+0000 7f02964ce700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0290100710 0x7f029019b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:40.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.122+0000 7f02964ce700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0290101100 0x7f029019de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:40.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.122+0000 7f02964ce700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029019e5c0 con 0x7f02900ffd50 2024-02-21T23:46:40.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.122+0000 7f02964ce700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029019e730 con 0x7f0290100710 2024-02-21T23:46:40.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.122+0000 7f02964ce700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029019e8f0 con 0x7f0290101100 2024-02-21T23:46:40.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.122+0000 7f028ffff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02900ffd50 0x7f02900ff760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:40.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.122+0000 7f028ffff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02900ffd50 0x7f02900ff760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49226/0 (socket says 172.21.15.90:49226) 2024-02-21T23:46:40.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.122+0000 7f028ffff700 1 -- 172.21.15.90:0/1979871382 learned_addr learned my addr 172.21.15.90:0/1979871382 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:40.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.122+0000 7f028f7fe700 1 --2- 172.21.15.90:0/1979871382 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0290100710 0x7f029019b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:40.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.122+0000 7f0294a6b700 1 --2- 172.21.15.90:0/1979871382 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0290101100 0x7f029019de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:40.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.123+0000 7f028ffff700 1 -- 172.21.15.90:0/1979871382 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0290101100 msgr2=0x7f029019de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:40.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.123+0000 7f028ffff700 1 --2- 172.21.15.90:0/1979871382 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0290101100 0x7f029019de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.123+0000 7f028ffff700 1 -- 172.21.15.90:0/1979871382 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0290100710 msgr2=0x7f029019b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:40.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.123+0000 7f028ffff700 1 --2- 172.21.15.90:0/1979871382 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0290100710 0x7f029019b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.123+0000 7f028ffff700 1 -- 172.21.15.90:0/1979871382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02800095f0 con 0x7f02900ffd50 2024-02-21T23:46:40.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.123+0000 7f028f7fe700 1 --2- 172.21.15.90:0/1979871382 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0290100710 0x7f029019b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:46:40.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.123+0000 7f028ffff700 1 --2- 172.21.15.90:0/1979871382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02900ffd50 0x7f02900ff760 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f02800095c0 tx=0x7f028000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:40.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.123+0000 7f028d7fa700 1 -- 172.21.15.90:0/1979871382 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028001d070 con 0x7f02900ffd50 2024-02-21T23:46:40.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.123+0000 7f028d7fa700 1 -- 172.21.15.90:0/1979871382 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f028000fb80 con 0x7f02900ffd50 2024-02-21T23:46:40.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.123+0000 7f02964ce700 1 -- 172.21.15.90:0/1979871382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f029019eb70 con 0x7f02900ffd50 2024-02-21T23:46:40.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.123+0000 7f028d7fa700 1 -- 172.21.15.90:0/1979871382 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0280022ea0 con 0x7f02900ffd50 2024-02-21T23:46:40.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.123+0000 7f02964ce700 1 -- 172.21.15.90:0/1979871382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f029019ef90 con 0x7f02900ffd50 2024-02-21T23:46:40.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.124+0000 7f02964ce700 1 -- 172.21.15.90:0/1979871382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0290106f40 con 0x7f02900ffd50 2024-02-21T23:46:40.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.125+0000 7f028d7fa700 1 -- 172.21.15.90:0/1979871382 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7f0280004830 con 0x7f02900ffd50 2024-02-21T23:46:40.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.125+0000 7f028d7fa700 1 --2- 172.21.15.90:0/1979871382 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f02780a1120 0x7f02780a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:40.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.125+0000 7f028f7fe700 1 --2- 172.21.15.90:0/1979871382 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f02780a1120 0x7f02780a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:40.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.125+0000 7f028d7fa700 1 -- 172.21.15.90:0/1979871382 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f02800cdf80 con 0x7f02900ffd50 2024-02-21T23:46:40.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.126+0000 7f028f7fe700 1 --2- 172.21.15.90:0/1979871382 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f02780a1120 0x7f02780a35e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0284005fd0 tx=0x7f0284005ee0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:40.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.129+0000 7f028d7fa700 1 -- 172.21.15.90:0/1979871382 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f02800975b0 con 0x7f02900ffd50 2024-02-21T23:46:40.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.371+0000 7f02964ce700 1 -- 172.21.15.90:0/1979871382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f029002d0e0 con 0x7f02900ffd50 2024-02-21T23:46:40.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.371+0000 7f028d7fa700 1 -- 172.21.15.90:0/1979871382 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+10816 (secure 0 0 0) 0x7f028009abd0 con 0x7f02900ffd50 2024-02-21T23:46:40.356 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:46:40.356 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/1591404300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1591404300},{"type":"v1","addr":"172.21.15.90:6835","nonce":1591404300}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/1441666263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":1441666263},{"type":"v1","addr":"172.21.15.90:6837","nonce":1441666263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14607,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6839/732535159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":732535159},{"type":"v1","addr":"172.21.15.90:6839","nonce":732535159}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14619,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/3572584154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":3572584154},{"type":"v1","addr":"172.21.15.90:6841","nonce":3572584154}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1403694104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1403694104},{"type":"v1","addr":"172.21.15.178:6833","nonce":1403694104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/2956287083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":2956287083},{"type":"v1","addr":"172.21.15.176:6833","nonce":2956287083}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/3430010314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":3430010314},{"type":"v1","addr":"172.21.15.176:6835","nonce":3430010314}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/1260308729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1260308729},{"type":"v1","addr":"172.21.15.178:6835","nonce":1260308729}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/4166803229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":4166803229},{"type":"v1","addr":"172.21.15.178:6837","nonce":4166803229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24458,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/4127802836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":4127802836},{"type":"v1","addr":"172.21.15.176:6837","nonce":4127802836}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24470,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6839/3726236229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":3726236229},{"type":"v1","addr":"172.21.15.178:6839","nonce":3726236229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:46:34.442043+0000","modified":"2024-02-21T23:46:38.878999+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.176:6839/875627757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6838","nonce":875627757},{"type":"v1","addr":"172.21.15.176:6839","nonce":875627757}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-02-21T23:46:40.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.374+0000 7f02964ce700 1 -- 172.21.15.90:0/1979871382 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f02780a1120 msgr2=0x7f02780a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:40.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.374+0000 7f02964ce700 1 --2- 172.21.15.90:0/1979871382 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f02780a1120 0x7f02780a35e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0284005fd0 tx=0x7f0284005ee0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.374+0000 7f02964ce700 1 -- 172.21.15.90:0/1979871382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02900ffd50 msgr2=0x7f02900ff760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:40.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.374+0000 7f02964ce700 1 --2- 172.21.15.90:0/1979871382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02900ffd50 0x7f02900ff760 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f02800095c0 tx=0x7f028000f690 comp rx=0 tx=0).stop 2024-02-21T23:46:40.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.374+0000 7f02964ce700 1 -- 172.21.15.90:0/1979871382 shutdown_connections 2024-02-21T23:46:40.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.374+0000 7f02964ce700 1 --2- 172.21.15.90:0/1979871382 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f02780a1120 0x7f02780a35e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.374+0000 7f02964ce700 1 --2- 172.21.15.90:0/1979871382 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0290101100 0x7f029019de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.374+0000 7f02964ce700 1 --2- 172.21.15.90:0/1979871382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02900ffd50 0x7f02900ff760 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.374+0000 7f02964ce700 1 --2- 172.21.15.90:0/1979871382 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0290100710 0x7f029019b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.374+0000 7f02964ce700 1 -- 172.21.15.90:0/1979871382 >> 172.21.15.90:0/1979871382 conn(0x7f02900fa7e0 msgr2=0x7f0290104080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:40.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.374+0000 7f02964ce700 1 -- 172.21.15.90:0/1979871382 shutdown_connections 2024-02-21T23:46:40.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.374+0000 7f02964ce700 1 -- 172.21.15.90:0/1979871382 wait complete. 2024-02-21T23:46:40.360 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 17 2024-02-21T23:46:40.371 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:46:40.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.509+0000 7f4b0885c700 1 -- 172.21.15.90:0/801783536 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b04101680 msgr2=0x7f4b04107f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:40.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.509+0000 7f4b0885c700 1 --2- 172.21.15.90:0/801783536 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b04101680 0x7f4b04107f30 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4afc00b0d0 tx=0x7f4afc00b3e0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.509+0000 7f4b0885c700 1 -- 172.21.15.90:0/801783536 shutdown_connections 2024-02-21T23:46:40.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.509+0000 7f4b0885c700 1 --2- 172.21.15.90:0/801783536 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b04101680 0x7f4b04107f30 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.509+0000 7f4b0885c700 1 --2- 172.21.15.90:0/801783536 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b040ffaf0 0x7f4b040fff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.509+0000 7f4b0885c700 1 --2- 172.21.15.90:0/801783536 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4b04100ce0 0x7f4b04101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.509+0000 7f4b0885c700 1 -- 172.21.15.90:0/801783536 >> 172.21.15.90:0/801783536 conn(0x7f4b040fb060 msgr2=0x7f4b040fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:40.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.509+0000 7f4b0885c700 1 -- 172.21.15.90:0/801783536 shutdown_connections 2024-02-21T23:46:40.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.509+0000 7f4b0885c700 1 -- 172.21.15.90:0/801783536 wait complete. 2024-02-21T23:46:40.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.510+0000 7f4b0885c700 1 Processor -- start 2024-02-21T23:46:40.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.510+0000 7f4b0885c700 1 -- start start 2024-02-21T23:46:40.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.510+0000 7f4b0885c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b040ffaf0 0x7f4b04195bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:40.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.510+0000 7f4b0885c700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4b04100ce0 0x7f4b04196110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:40.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.510+0000 7f4b0885c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b04101680 0x7f4b0419d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:40.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.510+0000 7f4b0885c700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0419d700 con 0x7f4b040ffaf0 2024-02-21T23:46:40.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.510+0000 7f4b0885c700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0419d840 con 0x7f4b04101680 2024-02-21T23:46:40.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.510+0000 7f4b0885c700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0419d9d0 con 0x7f4b04100ce0 2024-02-21T23:46:40.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.510+0000 7f4b0259c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b040ffaf0 0x7f4b04195bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:40.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.511+0000 7f4b0259c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b040ffaf0 0x7f4b04195bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49248/0 (socket says 172.21.15.90:49248) 2024-02-21T23:46:40.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.511+0000 7f4b0259c700 1 -- 172.21.15.90:0/2635142482 learned_addr learned my addr 172.21.15.90:0/2635142482 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:40.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.511+0000 7f4b01d9b700 1 --2- 172.21.15.90:0/2635142482 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4b04100ce0 0x7f4b04196110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:40.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.511+0000 7f4b0259c700 1 -- 172.21.15.90:0/2635142482 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4b04100ce0 msgr2=0x7f4b04196110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:40.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.511+0000 7f4b0259c700 1 --2- 172.21.15.90:0/2635142482 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4b04100ce0 0x7f4b04196110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.511+0000 7f4b0259c700 1 -- 172.21.15.90:0/2635142482 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b04101680 msgr2=0x7f4b0419d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:40.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.511+0000 7f4b0259c700 1 --2- 172.21.15.90:0/2635142482 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b04101680 0x7f4b0419d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.511+0000 7f4b0259c700 1 -- 172.21.15.90:0/2635142482 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4afc009d00 con 0x7f4b040ffaf0 2024-02-21T23:46:40.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.511+0000 7f4b0259c700 1 --2- 172.21.15.90:0/2635142482 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b040ffaf0 0x7f4b04195bd0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4af400b500 tx=0x7f4af400b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:40.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.511+0000 7f4af37fe700 1 -- 172.21.15.90:0/2635142482 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af4011830 con 0x7f4b040ffaf0 2024-02-21T23:46:40.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.511+0000 7f4b0885c700 1 -- 172.21.15.90:0/2635142482 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b0419dcb0 con 0x7f4b040ffaf0 2024-02-21T23:46:40.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.511+0000 7f4af37fe700 1 -- 172.21.15.90:0/2635142482 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4af4011e50 con 0x7f4b040ffaf0 2024-02-21T23:46:40.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.511+0000 7f4af37fe700 1 -- 172.21.15.90:0/2635142482 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af400f2e0 con 0x7f4b040ffaf0 2024-02-21T23:46:40.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.511+0000 7f4b0885c700 1 -- 172.21.15.90:0/2635142482 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b0419e1c0 con 0x7f4b040ffaf0 2024-02-21T23:46:40.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.512+0000 7f4b0885c700 1 -- 172.21.15.90:0/2635142482 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b04106ea0 con 0x7f4b040ffaf0 2024-02-21T23:46:40.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.513+0000 7f4af37fe700 1 -- 172.21.15.90:0/2635142482 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7f4af4010580 con 0x7f4b040ffaf0 2024-02-21T23:46:40.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.514+0000 7f4af37fe700 1 --2- 172.21.15.90:0/2635142482 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f4aec0a1120 0x7f4aec0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:40.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.514+0000 7f4b01d9b700 1 --2- 172.21.15.90:0/2635142482 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f4aec0a1120 0x7f4aec0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:40.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.514+0000 7f4af37fe700 1 -- 172.21.15.90:0/2635142482 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f4af40cc640 con 0x7f4b040ffaf0 2024-02-21T23:46:40.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.514+0000 7f4b01d9b700 1 --2- 172.21.15.90:0/2635142482 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f4aec0a1120 0x7f4aec0a35e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4af8005950 tx=0x7f4af8005cf0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:40.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.518+0000 7f4af37fe700 1 -- 172.21.15.90:0/2635142482 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f4af4095b80 con 0x7f4b040ffaf0 2024-02-21T23:46:40.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.719+0000 7f4b0885c700 1 -- 172.21.15.90:0/2635142482 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b040679a0 con 0x7f4b040ffaf0 2024-02-21T23:46:40.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.720+0000 7f4af37fe700 1 -- 172.21.15.90:0/2635142482 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23925 (secure 0 0 0) 0x7f4af40991a0 con 0x7f4b040ffaf0 2024-02-21T23:46:40.705 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:46:40.705 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":85,"fsid":"7ed81218-d112-11ee-95bd-87774f69a715","created":"2024-02-21T23:40:41.785526+0000","modified":"2024-02-21T23:46:38.878698+0000","last_up_change":"2024-02-21T23:46:00.860501+0000","last_in_change":"2024-02-21T23:45:44.423815+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:43:21.675789+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:46:32.337047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-21T23:46:33.622440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-21T23:46:36.686123+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"77878890-d9a5-4d5b-a943-fec29ee0446e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6803","nonce":885319826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6805","nonce":885319826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6809","nonce":885319826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6807","nonce":885319826}]},"public_addr":"172.21.15.90:6803/885319826","cluster_addr":"172.21.15.90:6805/885319826","heartbeat_back_addr":"172.21.15.90:6809/885319826","heartbeat_front_addr":"172.21.15.90:6807/885319826","state":["exists","up"]},{"osd":1,"uuid":"7d6ea950-7a57-4f13-ab38-10a658912783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6801","nonce":4085240925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6803","nonce":4085240925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6807","nonce":4085240925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6805","nonce":4085240925}]},"public_addr":"172.21.15.176:6801/4085240925","cluster_addr":"172.21.15.176:6803/4085240925","heartbeat_back_addr":"172.21.15.176:6807/4085240925","heartbeat_front_addr":"172.21.15.176:6805/4085240925","state":["exists","up"]},{"osd":2,"uuid":"b83486bb-27f2-4d56-97fb-fb336fec4b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6801","nonce":3597250789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6803","nonce":3597250789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6807","nonce":3597250789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6805","nonce":3597250789}]},"public_addr":"172.21.15.178:6801/3597250789","cluster_addr":"172.21.15.178:6803/3597250789","heartbeat_back_addr":"172.21.15.178:6807/3597250789","heartbeat_front_addr":"172.21.15.178:6805/3597250789","state":["exists","up"]},{"osd":3,"uuid":"870e5bb6-eed5-4a0a-8e03-38de4f816b44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6811","nonce":3299043821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6813","nonce":3299043821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6817","nonce":3299043821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6815","nonce":3299043821}]},"public_addr":"172.21.15.90:6811/3299043821","cluster_addr":"172.21.15.90:6813/3299043821","heartbeat_back_addr":"172.21.15.90:6817/3299043821","heartbeat_front_addr":"172.21.15.90:6815/3299043821","state":["exists","up"]},{"osd":4,"uuid":"cb9f7f48-76e0-4368-ab87-2a09a8487ccd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6809","nonce":3548915687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6811","nonce":3548915687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6815","nonce":3548915687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6813","nonce":3548915687}]},"public_addr":"172.21.15.176:6809/3548915687","cluster_addr":"172.21.15.176:6811/3548915687","heartbeat_back_addr":"172.21.15.176:6815/3548915687","heartbeat_front_addr":"172.21.15.176:6813/3548915687","state":["exists","up"]},{"osd":5,"uuid":"54749022-c50d-4831-a1d6-8412b59c8a01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6809","nonce":286274518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6811","nonce":286274518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6815","nonce":286274518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6813","nonce":286274518}]},"public_addr":"172.21.15.178:6809/286274518","cluster_addr":"172.21.15.178:6811/286274518","heartbeat_back_addr":"172.21.15.178:6815/286274518","heartbeat_front_addr":"172.21.15.178:6813/286274518","state":["exists","up"]},{"osd":6,"uuid":"09bb5393-2efd-4a63-b0e8-99bef5bced13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6819","nonce":67640620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6821","nonce":67640620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6825","nonce":67640620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6823","nonce":67640620}]},"public_addr":"172.21.15.90:6819/67640620","cluster_addr":"172.21.15.90:6821/67640620","heartbeat_back_addr":"172.21.15.90:6825/67640620","heartbeat_front_addr":"172.21.15.90:6823/67640620","state":["exists","up"]},{"osd":7,"uuid":"7a859e69-6b63-467b-a3b3-99247d66f5c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6817","nonce":3998561598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6819","nonce":3998561598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6823","nonce":3998561598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6821","nonce":3998561598}]},"public_addr":"172.21.15.176:6817/3998561598","cluster_addr":"172.21.15.176:6819/3998561598","heartbeat_back_addr":"172.21.15.176:6823/3998561598","heartbeat_front_addr":"172.21.15.176:6821/3998561598","state":["exists","up"]},{"osd":8,"uuid":"bc23144e-22a5-4894-a8b5-9323383574d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6817","nonce":490229963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6819","nonce":490229963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6823","nonce":490229963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6821","nonce":490229963}]},"public_addr":"172.21.15.178:6817/490229963","cluster_addr":"172.21.15.178:6819/490229963","heartbeat_back_addr":"172.21.15.178:6823/490229963","heartbeat_front_addr":"172.21.15.178:6821/490229963","state":["exists","up"]},{"osd":9,"uuid":"0c2cecd4-55de-4fc4-a6f4-ca97c8460444","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6827","nonce":2061394761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6829","nonce":2061394761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6833","nonce":2061394761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6831","nonce":2061394761}]},"public_addr":"172.21.15.90:6827/2061394761","cluster_addr":"172.21.15.90:6829/2061394761","heartbeat_back_addr":"172.21.15.90:6833/2061394761","heartbeat_front_addr":"172.21.15.90:6831/2061394761","state":["exists","up"]},{"osd":10,"uuid":"fa04232b-b1ec-447a-8e94-7c9bf26d0291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6825","nonce":639411757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6827","nonce":639411757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6831","nonce":639411757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6829","nonce":639411757}]},"public_addr":"172.21.15.176:6825/639411757","cluster_addr":"172.21.15.176:6827/639411757","heartbeat_back_addr":"172.21.15.176:6831/639411757","heartbeat_front_addr":"172.21.15.176:6829/639411757","state":["exists","up"]},{"osd":11,"uuid":"7282e294-64fc-4856-9c74-86115f69408c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6825","nonce":4144536256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6827","nonce":4144536256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6831","nonce":4144536256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6829","nonce":4144536256}]},"public_addr":"172.21.15.178:6825/4144536256","cluster_addr":"172.21.15.178:6827/4144536256","heartbeat_back_addr":"172.21.15.178:6831/4144536256","heartbeat_front_addr":"172.21.15.178:6829/4144536256","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:42:44.280345+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:01.235630+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:19.408896+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:36.201749+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:53.401854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:44:11.342158+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:44:29.324502+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:44:46.821595+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:04.642344+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:23.019004+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:41.140149+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:59.401951+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/159655312":"2024-02-22T23:41:33.360021+0000","172.21.15.90:6801/1385551604":"2024-02-22T23:41:33.360021+0000","172.21.15.90:0/1452574770":"2024-02-22T23:41:33.360021+0000","172.21.15.90:0/3815292343":"2024-02-22T23:41:11.484145+0000","172.21.15.90:6800/1385551604":"2024-02-22T23:41:33.360021+0000","172.21.15.90:6801/4247695574":"2024-02-22T23:41:11.484145+0000","172.21.15.90:0/1328266622":"2024-02-22T23:41:11.484145+0000","172.21.15.90:6800/4247695574":"2024-02-22T23:41:11.484145+0000","172.21.15.90:0/2569421522":"2024-02-22T23:41:11.484145+0000","172.21.15.90:0/3019701027":"2024-02-22T23:41:33.360021+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:46:40.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.723+0000 7f4b0885c700 1 -- 172.21.15.90:0/2635142482 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f4aec0a1120 msgr2=0x7f4aec0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:40.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.723+0000 7f4b0885c700 1 --2- 172.21.15.90:0/2635142482 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f4aec0a1120 0x7f4aec0a35e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4af8005950 tx=0x7f4af8005cf0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.723+0000 7f4b0885c700 1 -- 172.21.15.90:0/2635142482 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b040ffaf0 msgr2=0x7f4b04195bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:40.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.723+0000 7f4b0885c700 1 --2- 172.21.15.90:0/2635142482 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b040ffaf0 0x7f4b04195bd0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4af400b500 tx=0x7f4af400b8c0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.723+0000 7f4b0885c700 1 -- 172.21.15.90:0/2635142482 shutdown_connections 2024-02-21T23:46:40.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.723+0000 7f4b0885c700 1 --2- 172.21.15.90:0/2635142482 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f4aec0a1120 0x7f4aec0a35e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.723+0000 7f4b0885c700 1 --2- 172.21.15.90:0/2635142482 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b04101680 0x7f4b0419d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.723+0000 7f4b0885c700 1 --2- 172.21.15.90:0/2635142482 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b040ffaf0 0x7f4b04195bd0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.723+0000 7f4b0885c700 1 --2- 172.21.15.90:0/2635142482 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4b04100ce0 0x7f4b04196110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.723+0000 7f4b0885c700 1 -- 172.21.15.90:0/2635142482 >> 172.21.15.90:0/2635142482 conn(0x7f4b040fb060 msgr2=0x7f4b04103fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:40.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.723+0000 7f4b0885c700 1 -- 172.21.15.90:0/2635142482 shutdown_connections 2024-02-21T23:46:40.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.723+0000 7f4b0885c700 1 -- 172.21.15.90:0/2635142482 wait complete. 2024-02-21T23:46:40.718 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:40 smithi090 ceph-mon[43385]: pgmap v228: 129 pgs: 3 creating+peering, 28 unknown, 98 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 11 op/s 2024-02-21T23:46:40.718 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:40 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/725991941' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-21T23:46:40.719 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:40 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/202032530' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-21T23:46:40.719 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:40 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/1979871382' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-21T23:46:40.720 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:46:34.442043+0000', 'modified': '2024-02-21T23:46:38.878999+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24500}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24500': {'gid': 24500, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.176:6839/875627757', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6838', 'nonce': 875627757}, {'type': 'v1', 'addr': '172.21.15.176:6839', 'nonce': 875627757}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2024-02-21T23:46:40.720 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-21T23:46:40.720 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-02-21T23:46:40.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:40 smithi176 ceph-mon[47172]: pgmap v228: 129 pgs: 3 creating+peering, 28 unknown, 98 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 11 op/s 2024-02-21T23:46:40.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:40 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/725991941' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-21T23:46:40.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:40 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/202032530' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-21T23:46:40.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:40 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/1979871382' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-21T23:46:40.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.858+0000 7fb832b5e700 1 -- 172.21.15.90:0/3883256308 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb82c101f90 msgr2=0x7fb82c1023f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:40.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.858+0000 7fb832b5e700 1 --2- 172.21.15.90:0/3883256308 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb82c101f90 0x7fb82c1023f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb818009990 tx=0x7fb818009ca0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.858+0000 7fb832b5e700 1 -- 172.21.15.90:0/3883256308 shutdown_connections 2024-02-21T23:46:40.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.858+0000 7fb832b5e700 1 --2- 172.21.15.90:0/3883256308 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb82c102930 0x7fb82c1091e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.858+0000 7fb832b5e700 1 --2- 172.21.15.90:0/3883256308 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb82c0fefc0 0x7fb82c0ff3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.858+0000 7fb832b5e700 1 --2- 172.21.15.90:0/3883256308 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb82c101f90 0x7fb82c1023f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.858+0000 7fb832b5e700 1 -- 172.21.15.90:0/3883256308 >> 172.21.15.90:0/3883256308 conn(0x7fb82c0fa8f0 msgr2=0x7fb82c0fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:40.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.858+0000 7fb832b5e700 1 -- 172.21.15.90:0/3883256308 shutdown_connections 2024-02-21T23:46:40.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.858+0000 7fb832b5e700 1 -- 172.21.15.90:0/3883256308 wait complete. 2024-02-21T23:46:40.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.859+0000 7fb832b5e700 1 Processor -- start 2024-02-21T23:46:40.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.859+0000 7fb832b5e700 1 -- start start 2024-02-21T23:46:40.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.859+0000 7fb832b5e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb82c0fefc0 0x7fb82c196c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:40.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.859+0000 7fb832b5e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb82c101f90 0x7fb82c197160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:40.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.859+0000 7fb832b5e700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb82c102930 0x7fb82c19e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:40.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.859+0000 7fb832b5e700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb82c19e750 con 0x7fb82c0fefc0 2024-02-21T23:46:40.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.859+0000 7fb832b5e700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb82c19e890 con 0x7fb82c101f90 2024-02-21T23:46:40.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.859+0000 7fb832b5e700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb82c19ea20 con 0x7fb82c102930 2024-02-21T23:46:40.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.859+0000 7fb8308fa700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb82c0fefc0 0x7fb82c196c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:40.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.859+0000 7fb8308fa700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb82c0fefc0 0x7fb82c196c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49268/0 (socket says 172.21.15.90:49268) 2024-02-21T23:46:40.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.859+0000 7fb8308fa700 1 -- 172.21.15.90:0/1085379517 learned_addr learned my addr 172.21.15.90:0/1085379517 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:40.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.859+0000 7fb8310fb700 1 --2- 172.21.15.90:0/1085379517 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb82c102930 0x7fb82c19e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:40.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.859+0000 7fb82bfff700 1 --2- 172.21.15.90:0/1085379517 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb82c101f90 0x7fb82c197160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:40.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.859+0000 7fb8308fa700 1 -- 172.21.15.90:0/1085379517 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb82c102930 msgr2=0x7fb82c19e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:40.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.859+0000 7fb8308fa700 1 --2- 172.21.15.90:0/1085379517 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb82c102930 0x7fb82c19e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.859+0000 7fb8308fa700 1 -- 172.21.15.90:0/1085379517 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb82c101f90 msgr2=0x7fb82c197160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:40.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.859+0000 7fb8308fa700 1 --2- 172.21.15.90:0/1085379517 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb82c101f90 0x7fb82c197160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.859+0000 7fb8308fa700 1 -- 172.21.15.90:0/1085379517 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8180095f0 con 0x7fb82c0fefc0 2024-02-21T23:46:40.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.859+0000 7fb82bfff700 1 --2- 172.21.15.90:0/1085379517 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb82c101f90 0x7fb82c197160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:46:40.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.859+0000 7fb8308fa700 1 --2- 172.21.15.90:0/1085379517 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb82c0fefc0 0x7fb82c196c20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb82000d980 tx=0x7fb82000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:40.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.860+0000 7fb829ffb700 1 -- 172.21.15.90:0/1085379517 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb820004020 con 0x7fb82c0fefc0 2024-02-21T23:46:40.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.860+0000 7fb832b5e700 1 -- 172.21.15.90:0/1085379517 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb82c19ed00 con 0x7fb82c0fefc0 2024-02-21T23:46:40.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.860+0000 7fb829ffb700 1 -- 172.21.15.90:0/1085379517 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb820009730 con 0x7fb82c0fefc0 2024-02-21T23:46:40.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.860+0000 7fb829ffb700 1 -- 172.21.15.90:0/1085379517 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb820003a30 con 0x7fb82c0fefc0 2024-02-21T23:46:40.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.860+0000 7fb832b5e700 1 -- 172.21.15.90:0/1085379517 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb82c19f1b0 con 0x7fb82c0fefc0 2024-02-21T23:46:40.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.862+0000 7fb829ffb700 1 -- 172.21.15.90:0/1085379517 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7fb8200098a0 con 0x7fb82c0fefc0 2024-02-21T23:46:40.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.862+0000 7fb829ffb700 1 --2- 172.21.15.90:0/1085379517 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fb81c0a1120 0x7fb81c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:40.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.862+0000 7fb82bfff700 1 --2- 172.21.15.90:0/1085379517 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fb81c0a1120 0x7fb81c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:40.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.862+0000 7fb829ffb700 1 -- 172.21.15.90:0/1085379517 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fb8200cc780 con 0x7fb82c0fefc0 2024-02-21T23:46:40.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.863+0000 7fb82bfff700 1 --2- 172.21.15.90:0/1085379517 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fb81c0a1120 0x7fb81c0a35e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb8180037f0 tx=0x7fb818005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:40.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.863+0000 7fb83ecf1b80 1 Processor -- start 2024-02-21T23:46:40.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.863+0000 7fb83ecf1b80 1 -- start start 2024-02-21T23:46:40.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.863+0000 7fb83ecf1b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558a0e30ad40 0x558a0e30b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:40.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.863+0000 7fb83ecf1b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558a0e30b6a0 0x558a0e30bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:40.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.863+0000 7fb83ecf1b80 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558a0e30c590 0x558a0e30ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:40.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.863+0000 7fb83ecf1b80 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a0e263910 con 0x558a0e30c590 2024-02-21T23:46:40.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.863+0000 7fb83ecf1b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a0e2612d0 con 0x558a0e30ad40 2024-02-21T23:46:40.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.863+0000 7fb83ecf1b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a0e25cbc0 con 0x558a0e30b6a0 2024-02-21T23:46:40.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.863+0000 7fb8310fb700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558a0e30c590 0x558a0e30ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:40.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.863+0000 7fb8310fb700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558a0e30c590 0x558a0e30ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49278/0 (socket says 172.21.15.90:49278) 2024-02-21T23:46:40.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.863+0000 7fb8310fb700 1 -- 172.21.15.90:0/1489943225 learned_addr learned my addr 172.21.15.90:0/1489943225 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:40.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.863+0000 7fb8308fa700 1 --2- 172.21.15.90:0/1489943225 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558a0e30ad40 0x558a0e30b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:40.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.864+0000 7fb82bfff700 1 --2- 172.21.15.90:0/1489943225 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558a0e30b6a0 0x558a0e30bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:40.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.864+0000 7fb8310fb700 1 -- 172.21.15.90:0/1489943225 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558a0e30b6a0 msgr2=0x558a0e30bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:46:40.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.864+0000 7fb8310fb700 1 --2- 172.21.15.90:0/1489943225 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558a0e30b6a0 0x558a0e30bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.864+0000 7fb8310fb700 1 -- 172.21.15.90:0/1489943225 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558a0e30ad40 msgr2=0x558a0e30b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:46:40.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.864+0000 7fb8310fb700 1 --2- 172.21.15.90:0/1489943225 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558a0e30ad40 0x558a0e30b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.864+0000 7fb8310fb700 1 -- 172.21.15.90:0/1489943225 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558a0e08ab10 con 0x558a0e30c590 2024-02-21T23:46:40.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.864+0000 7fb82bfff700 1 --2- 172.21.15.90:0/1489943225 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558a0e30b6a0 0x558a0e30bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:46:40.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.864+0000 7fb8310fb700 1 --2- 172.21.15.90:0/1489943225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558a0e30c590 0x558a0e30ca10 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb8240098f0 tx=0x7fb824009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a0e3873ef02279 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:40.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.864+0000 7fb816ffd700 1 -- 172.21.15.90:0/1489943225 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb824019070 con 0x558a0e30c590 2024-02-21T23:46:40.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.864+0000 7fb816ffd700 1 -- 172.21.15.90:0/1489943225 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb82400f040 con 0x558a0e30c590 2024-02-21T23:46:40.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.864+0000 7fb816ffd700 1 -- 172.21.15.90:0/1489943225 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb824007be0 con 0x558a0e30c590 2024-02-21T23:46:40.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.864+0000 7fb83ecf1b80 1 -- 172.21.15.90:0/1489943225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558a0e30c590 msgr2=0x558a0e30ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:40.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.864+0000 7fb83ecf1b80 1 --2- 172.21.15.90:0/1489943225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558a0e30c590 0x558a0e30ca10 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb8240098f0 tx=0x7fb824009c00 comp rx=0 tx=0).stop 2024-02-21T23:46:40.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.865+0000 7fb83ecf1b80 1 -- 172.21.15.90:0/1489943225 shutdown_connections 2024-02-21T23:46:40.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.865+0000 7fb83ecf1b80 1 --2- 172.21.15.90:0/1489943225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558a0e30c590 0x558a0e30ca10 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.865+0000 7fb83ecf1b80 1 --2- 172.21.15.90:0/1489943225 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558a0e30ad40 0x558a0e30b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.865+0000 7fb83ecf1b80 1 --2- 172.21.15.90:0/1489943225 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558a0e30b6a0 0x558a0e30bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.865+0000 7fb83ecf1b80 1 -- 172.21.15.90:0/1489943225 >> 172.21.15.90:0/1489943225 conn(0x558a0e26a620 msgr2=0x558a0e305d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:40.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.865+0000 7fb83ecf1b80 1 -- 172.21.15.90:0/1489943225 shutdown_connections 2024-02-21T23:46:40.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.865+0000 7fb83ecf1b80 1 -- 172.21.15.90:0/1489943225 wait complete. 2024-02-21T23:46:40.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.865+0000 7fb83ecf1b80 1 -- start start 2024-02-21T23:46:40.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.865+0000 7fb83ecf1b80 1 Processor -- start 2024-02-21T23:46:40.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.865+0000 7fb83ecf1b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558a0e30ad40 0x558a0e314a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:40.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.865+0000 7fb83ecf1b80 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558a0e30b6a0 0x558a0e314f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:40.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.865+0000 7fb83ecf1b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558a0e30c590 0x558a0e31bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:40.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.865+0000 7fb83ecf1b80 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a0e1f8d30 con 0x558a0e30b6a0 2024-02-21T23:46:40.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.865+0000 7fb83ecf1b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a0e1f89d0 con 0x558a0e30c590 2024-02-21T23:46:40.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.865+0000 7fb83ecf1b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a0e0e0650 con 0x558a0e30ad40 2024-02-21T23:46:40.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.865+0000 7fb83ecf1b80 10 client.0 _ll_register_callbacks cb 0x558a0e064350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:46:40.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.865+0000 7fb82bfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558a0e30b6a0 0x558a0e314f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:40.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.865+0000 7fb82bfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558a0e30b6a0 0x558a0e314f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49284/0 (socket says 172.21.15.90:49284) 2024-02-21T23:46:40.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.865+0000 7fb82bfff700 1 -- 172.21.15.90:0/2816673749 learned_addr learned my addr 172.21.15.90:0/2816673749 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:40.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.866+0000 7fb8308fa700 1 --2- 172.21.15.90:0/2816673749 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558a0e30ad40 0x558a0e314a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:40.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.866+0000 7fb82bfff700 1 -- 172.21.15.90:0/2816673749 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558a0e30ad40 msgr2=0x558a0e314a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:46:40.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.866+0000 7fb82bfff700 1 --2- 172.21.15.90:0/2816673749 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558a0e30ad40 0x558a0e314a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.866+0000 7fb82bfff700 1 -- 172.21.15.90:0/2816673749 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558a0e30c590 msgr2=0x558a0e31bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:46:40.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.866+0000 7fb82bfff700 1 --2- 172.21.15.90:0/2816673749 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558a0e30c590 0x558a0e31bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.866+0000 7fb82bfff700 1 -- 172.21.15.90:0/2816673749 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb824003c50 con 0x558a0e30b6a0 2024-02-21T23:46:40.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.866+0000 7fb8308fa700 1 --2- 172.21.15.90:0/2816673749 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558a0e30ad40 0x558a0e314a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:46:40.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.866+0000 7fb82bfff700 1 --2- 172.21.15.90:0/2816673749 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558a0e30b6a0 0x558a0e314f90 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb8180093d0 tx=0x7fb818009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ad3c99b61649faf server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:40.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.866+0000 7fb80bfff700 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-02-21T23:46:40.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.866+0000 7fb80bfff700 1 -- 172.21.15.90:0/2816673749 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb818003a90 con 0x558a0e30b6a0 2024-02-21T23:46:40.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.866+0000 7fb83ecf1b80 1 -- 172.21.15.90:0/2816673749 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558a0e2fd9f0 con 0x558a0e30b6a0 2024-02-21T23:46:40.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.866+0000 7fb80bfff700 1 -- 172.21.15.90:0/2816673749 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fb818003bf0 con 0x558a0e30b6a0 2024-02-21T23:46:40.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.866+0000 7fb80bfff700 1 -- 172.21.15.90:0/2816673749 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb818003db0 con 0x558a0e30b6a0 2024-02-21T23:46:40.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.866+0000 7fb83ecf1b80 10 client.14736 fetch_fsmap learned FSMap version 17 2024-02-21T23:46:40.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.866+0000 7fb83ecf1b80 1 -- 172.21.15.90:0/2816673749 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x558a0e222e50 con 0x558a0e30b6a0 2024-02-21T23:46:40.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.866+0000 7fb80bfff700 1 -- 172.21.15.90:0/2816673749 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb81803a080 con 0x558a0e30b6a0 2024-02-21T23:46:40.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.866+0000 7fb80bfff700 1 -- 172.21.15.90:0/2816673749 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 17) v1 ==== 7482+0+0 (secure 0 0 0) 0x7fb81802f8a0 con 0x558a0e30b6a0 2024-02-21T23:46:40.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.866+0000 7fb83ecf1b80 10 client.14736 fetch_fsmap finished waiting for FSMap version 17 2024-02-21T23:46:40.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.866+0000 7fb83ecf1b80 10 client.14736 resolve_mds: resolved name 'b' to daemon mds.b 2024-02-21T23:46:40.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.867+0000 7fb83ecf1b80 20 client.14736 populate_metadata read hostname 'smithi090' 2024-02-21T23:46:40.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.867+0000 7fb83ecf1b80 1 --2- 172.21.15.90:0/2816673749 >> [v2:172.21.15.176:6838/875627757,v1:172.21.15.176:6839/875627757] conn(0x558a0e31f600 0x558a0e324f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:40.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.867+0000 7fb83ecf1b80 4 client.14736 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2024-02-21T23:46:40.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.867+0000 7fb83ecf1b80 1 -- 172.21.15.90:0/2816673749 --> [v2:172.21.15.176:6838/875627757,v1:172.21.15.176:6839/875627757] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558a0e2557d0 con 0x558a0e31f600 2024-02-21T23:46:40.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.867+0000 7fb8308fa700 1 --2- 172.21.15.90:0/2816673749 >> [v2:172.21.15.176:6838/875627757,v1:172.21.15.176:6839/875627757] conn(0x558a0e31f600 0x558a0e324f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:40.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.867+0000 7fb80bfff700 10 client.14736 ms_handle_connect on v2:172.21.15.176:6838/875627757 2024-02-21T23:46:40.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.867+0000 7fb8308fa700 1 --2- 172.21.15.90:0/2816673749 >> [v2:172.21.15.176:6838/875627757,v1:172.21.15.176:6839/875627757] conn(0x558a0e31f600 0x558a0e324f30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1a90c79fc436067 server_cookie=1ee50c0e23614909 in_seq=0 out_seq=0 2024-02-21T23:46:40.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.868+0000 7fb80bfff700 1 -- 172.21.15.90:0/2816673749 <== mds.0 v2:172.21.15.176:6838/875627757 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x558a0e2557d0 con 0x558a0e31f600 2024-02-21T23:46:40.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.868+0000 7fb80bfff700 10 client.14736 handle_command_reply: tid=0 2024-02-21T23:46:40.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.868+0000 7fb83ecf1b80 1 client.14736 shutdown 2024-02-21T23:46:40.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.868+0000 7fb83ecf1b80 1 -- 172.21.15.90:0/2816673749 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558a0e30b6a0 msgr2=0x558a0e314f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:40.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.868+0000 7fb83ecf1b80 1 --2- 172.21.15.90:0/2816673749 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558a0e30b6a0 0x558a0e314f90 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb8180093d0 tx=0x7fb818009ca0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.868+0000 7fb83ecf1b80 1 -- 172.21.15.90:0/2816673749 shutdown_connections 2024-02-21T23:46:40.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.868+0000 7fb83ecf1b80 1 --2- 172.21.15.90:0/2816673749 >> [v2:172.21.15.176:6838/875627757,v1:172.21.15.176:6839/875627757] conn(0x558a0e31f600 0x558a0e324f30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.868+0000 7fb83ecf1b80 1 --2- 172.21.15.90:0/2816673749 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558a0e30c590 0x558a0e31bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.868+0000 7fb83ecf1b80 1 --2- 172.21.15.90:0/2816673749 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558a0e30ad40 0x558a0e314a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.868+0000 7fb83ecf1b80 1 --2- 172.21.15.90:0/2816673749 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558a0e30b6a0 0x558a0e314f90 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.868+0000 7fb80bfff700 0 client.14736 ms_handle_reset on v2:172.21.15.176:6838/875627757 2024-02-21T23:46:40.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.868+0000 7fb83ecf1b80 1 -- 172.21.15.90:0/2816673749 >> 172.21.15.90:0/2816673749 conn(0x558a0e26a620 msgr2=0x558a0e309830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:40.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.868+0000 7fb83ecf1b80 1 -- 172.21.15.90:0/2816673749 shutdown_connections 2024-02-21T23:46:40.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.868+0000 7fb83ecf1b80 1 -- 172.21.15.90:0/2816673749 wait complete. 2024-02-21T23:46:40.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.868+0000 7fb83ecf1b80 20 client.14736 trim_cache size 0 max 16384 2024-02-21T23:46:40.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.885+0000 7fb83ecf1b80 1 Processor -- start 2024-02-21T23:46:40.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.885+0000 7fb83ecf1b80 1 -- start start 2024-02-21T23:46:40.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.886+0000 7fb83ecf1b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558a0e30ad40 0x558a0e30b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:40.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.886+0000 7fb83ecf1b80 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558a0e30b6a0 0x558a0e319f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:40.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.886+0000 7fb83ecf1b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558a0e30c590 0x558a0e30c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:40.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.886+0000 7fb83ecf1b80 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a0e255ab0 con 0x558a0e30b6a0 2024-02-21T23:46:40.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.886+0000 7fb83ecf1b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a0e259820 con 0x558a0e30ad40 2024-02-21T23:46:40.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.886+0000 7fb83ecf1b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a0e1fbde0 con 0x558a0e30c590 2024-02-21T23:46:40.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.886+0000 7fb82bfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558a0e30b6a0 0x558a0e319f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:40.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.886+0000 7fb82bfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558a0e30b6a0 0x558a0e319f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49294/0 (socket says 172.21.15.90:49294) 2024-02-21T23:46:40.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.886+0000 7fb82bfff700 1 -- 172.21.15.90:0/3359036554 learned_addr learned my addr 172.21.15.90:0/3359036554 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:40.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.886+0000 7fb82bfff700 1 -- 172.21.15.90:0/3359036554 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558a0e30c590 msgr2=0x558a0e30c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:46:40.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.886+0000 7fb82bfff700 1 --2- 172.21.15.90:0/3359036554 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558a0e30c590 0x558a0e30c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.886+0000 7fb82bfff700 1 -- 172.21.15.90:0/3359036554 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558a0e30ad40 msgr2=0x558a0e30b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:46:40.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.886+0000 7fb82bfff700 1 --2- 172.21.15.90:0/3359036554 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558a0e30ad40 0x558a0e30b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.886+0000 7fb82bfff700 1 -- 172.21.15.90:0/3359036554 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558a0e0e2c80 con 0x558a0e30b6a0 2024-02-21T23:46:40.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.886+0000 7fb82bfff700 1 --2- 172.21.15.90:0/3359036554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558a0e30b6a0 0x558a0e319f80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb818009990 tx=0x7fb81800ff00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bbef7655dce2a9b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:40.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.886+0000 7fb8177fe700 1 -- 172.21.15.90:0/3359036554 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81802f450 con 0x558a0e30b6a0 2024-02-21T23:46:40.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.886+0000 7fb8177fe700 1 -- 172.21.15.90:0/3359036554 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558a0e2fd9f0 con 0x558a0e30b6a0 2024-02-21T23:46:40.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.886+0000 7fb8177fe700 1 -- 172.21.15.90:0/3359036554 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb818005310 con 0x558a0e30b6a0 2024-02-21T23:46:40.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.887+0000 7fb83ecf1b80 1 -- 172.21.15.90:0/3359036554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558a0e30b6a0 msgr2=0x558a0e319f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:40.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.887+0000 7fb83ecf1b80 1 --2- 172.21.15.90:0/3359036554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558a0e30b6a0 0x558a0e319f80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb818009990 tx=0x7fb81800ff00 comp rx=0 tx=0).stop 2024-02-21T23:46:40.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.887+0000 7fb83ecf1b80 1 -- 172.21.15.90:0/3359036554 shutdown_connections 2024-02-21T23:46:40.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.887+0000 7fb83ecf1b80 1 --2- 172.21.15.90:0/3359036554 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558a0e30c590 0x558a0e30c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.887+0000 7fb83ecf1b80 1 --2- 172.21.15.90:0/3359036554 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558a0e30ad40 0x558a0e30b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.887+0000 7fb83ecf1b80 1 --2- 172.21.15.90:0/3359036554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558a0e30b6a0 0x558a0e319f80 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.887+0000 7fb83ecf1b80 1 -- 172.21.15.90:0/3359036554 >> 172.21.15.90:0/3359036554 conn(0x558a0e26a620 msgr2=0x558a0e313f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:40.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.887+0000 7fb83ecf1b80 1 -- 172.21.15.90:0/3359036554 shutdown_connections 2024-02-21T23:46:40.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.887+0000 7fb83ecf1b80 1 -- 172.21.15.90:0/3359036554 wait complete. 2024-02-21T23:46:40.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.887+0000 7fb83ecf1b80 1 -- start start 2024-02-21T23:46:40.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.887+0000 7fb83ecf1b80 1 Processor -- start 2024-02-21T23:46:40.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.887+0000 7fb83ecf1b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558a0e30ad40 0x558a0e319b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:40.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.888+0000 7fb83ecf1b80 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558a0e30b6a0 0x558a0e318230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:40.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.888+0000 7fb83ecf1b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558a0e30c590 0x558a0e318770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:40.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.888+0000 7fb83ecf1b80 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a0e318cb0 con 0x558a0e30b6a0 2024-02-21T23:46:40.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.888+0000 7fb83ecf1b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a0e318e20 con 0x558a0e30ad40 2024-02-21T23:46:40.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.888+0000 7fb83ecf1b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a0e318fe0 con 0x558a0e30c590 2024-02-21T23:46:40.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.888+0000 7fb83ecf1b80 10 client.0 _ll_register_callbacks cb 0x558a0e064350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:46:40.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.888+0000 7fb82bfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558a0e30b6a0 0x558a0e318230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:40.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.888+0000 7fb82bfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558a0e30b6a0 0x558a0e318230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49306/0 (socket says 172.21.15.90:49306) 2024-02-21T23:46:40.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.888+0000 7fb82bfff700 1 -- 172.21.15.90:0/1299703384 learned_addr learned my addr 172.21.15.90:0/1299703384 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:40.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.888+0000 7fb82bfff700 1 -- 172.21.15.90:0/1299703384 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558a0e30c590 msgr2=0x558a0e318770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:46:40.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.888+0000 7fb82bfff700 1 --2- 172.21.15.90:0/1299703384 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558a0e30c590 0x558a0e318770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.888+0000 7fb82bfff700 1 -- 172.21.15.90:0/1299703384 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558a0e30ad40 msgr2=0x558a0e319b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:46:40.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.888+0000 7fb82bfff700 1 --2- 172.21.15.90:0/1299703384 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558a0e30ad40 0x558a0e319b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.888+0000 7fb82bfff700 1 -- 172.21.15.90:0/1299703384 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558a0e222e50 con 0x558a0e30b6a0 2024-02-21T23:46:40.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.888+0000 7fb82bfff700 1 --2- 172.21.15.90:0/1299703384 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558a0e30b6a0 0x558a0e318230 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb81800b540 tx=0x7fb81800b570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfe76475814e9036 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:40.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.888+0000 7fb80bfff700 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-02-21T23:46:40.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.888+0000 7fb80bfff700 1 -- 172.21.15.90:0/1299703384 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81802f660 con 0x558a0e30b6a0 2024-02-21T23:46:40.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.888+0000 7fb80bfff700 1 -- 172.21.15.90:0/1299703384 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fb81802f7c0 con 0x558a0e30b6a0 2024-02-21T23:46:40.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.888+0000 7fb83ecf1b80 1 -- 172.21.15.90:0/1299703384 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb82400f040 con 0x558a0e30b6a0 2024-02-21T23:46:40.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.888+0000 7fb80bfff700 1 -- 172.21.15.90:0/1299703384 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb81802f980 con 0x558a0e30b6a0 2024-02-21T23:46:40.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.888+0000 7fb83ecf1b80 10 client.14742 fetch_fsmap learned FSMap version 17 2024-02-21T23:46:40.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.888+0000 7fb83ecf1b80 1 -- 172.21.15.90:0/1299703384 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x558a0e088fe0 con 0x558a0e30b6a0 2024-02-21T23:46:40.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.888+0000 7fb80bfff700 1 -- 172.21.15.90:0/1299703384 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb82400f040 con 0x558a0e30b6a0 2024-02-21T23:46:40.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.889+0000 7fb80bfff700 1 -- 172.21.15.90:0/1299703384 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 17) v1 ==== 7482+0+0 (secure 0 0 0) 0x7fb818036a80 con 0x558a0e30b6a0 2024-02-21T23:46:40.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.889+0000 7fb83ecf1b80 10 client.14742 fetch_fsmap finished waiting for FSMap version 17 2024-02-21T23:46:40.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.889+0000 7fb83ecf1b80 10 client.14742 resolve_mds: resolved name 'b' to daemon mds.b 2024-02-21T23:46:40.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.889+0000 7fb83ecf1b80 20 client.14742 populate_metadata read hostname 'smithi090' 2024-02-21T23:46:40.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.889+0000 7fb83ecf1b80 1 --2- 172.21.15.90:0/1299703384 >> [v2:172.21.15.176:6838/875627757,v1:172.21.15.176:6839/875627757] conn(0x558a0e376820 0x558a0e373180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:40.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.889+0000 7fb83ecf1b80 4 client.14742 mds_command: new command op to 24500 tid=0[{"prefix": "status"}] 2024-02-21T23:46:40.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.889+0000 7fb83ecf1b80 1 -- 172.21.15.90:0/1299703384 --> [v2:172.21.15.176:6838/875627757,v1:172.21.15.176:6839/875627757] -- command(tid 0: {"prefix": "status"}) v1 -- 0x558a0e0abca0 con 0x558a0e376820 2024-02-21T23:46:40.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.889+0000 7fb8308fa700 1 --2- 172.21.15.90:0/1299703384 >> [v2:172.21.15.176:6838/875627757,v1:172.21.15.176:6839/875627757] conn(0x558a0e376820 0x558a0e373180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:40.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.889+0000 7fb80bfff700 10 client.14742 ms_handle_connect on v2:172.21.15.176:6838/875627757 2024-02-21T23:46:40.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.889+0000 7fb8308fa700 1 --2- 172.21.15.90:0/1299703384 >> [v2:172.21.15.176:6838/875627757,v1:172.21.15.176:6839/875627757] conn(0x558a0e376820 0x558a0e373180 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5dce9a37d6a4a302 server_cookie=2a526af0688ca9bb in_seq=0 out_seq=0 2024-02-21T23:46:40.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.890+0000 7fb80bfff700 1 -- 172.21.15.90:0/1299703384 <== mds.0 v2:172.21.15.176:6838/875627757 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x558a0e0abca0 con 0x558a0e376820 2024-02-21T23:46:40.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.890+0000 7fb80bfff700 10 client.14742 handle_command_reply: tid=0 2024-02-21T23:46:40.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.890+0000 7fb83ecf1b80 1 client.14742 shutdown 2024-02-21T23:46:40.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.890+0000 7fb83ecf1b80 1 -- 172.21.15.90:0/1299703384 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558a0e30b6a0 msgr2=0x558a0e318230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:40.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.890+0000 7fb83ecf1b80 1 --2- 172.21.15.90:0/1299703384 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558a0e30b6a0 0x558a0e318230 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb81800b540 tx=0x7fb81800b570 comp rx=0 tx=0).stop 2024-02-21T23:46:40.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.890+0000 7fb83ecf1b80 1 -- 172.21.15.90:0/1299703384 shutdown_connections 2024-02-21T23:46:40.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.890+0000 7fb83ecf1b80 1 --2- 172.21.15.90:0/1299703384 >> [v2:172.21.15.176:6838/875627757,v1:172.21.15.176:6839/875627757] conn(0x558a0e376820 0x558a0e373180 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.890+0000 7fb83ecf1b80 1 --2- 172.21.15.90:0/1299703384 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x558a0e30c590 0x558a0e318770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.890+0000 7fb80bfff700 0 client.14742 ms_handle_reset on v2:172.21.15.176:6838/875627757 2024-02-21T23:46:40.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.890+0000 7fb83ecf1b80 1 --2- 172.21.15.90:0/1299703384 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558a0e30ad40 0x558a0e319b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.890+0000 7fb83ecf1b80 1 --2- 172.21.15.90:0/1299703384 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558a0e30b6a0 0x558a0e318230 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.890+0000 7fb83ecf1b80 1 -- 172.21.15.90:0/1299703384 >> 172.21.15.90:0/1299703384 conn(0x558a0e26a620 msgr2=0x558a0e313f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:40.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.890+0000 7fb83ecf1b80 1 -- 172.21.15.90:0/1299703384 shutdown_connections 2024-02-21T23:46:40.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.890+0000 7fb83ecf1b80 1 -- 172.21.15.90:0/1299703384 wait complete. 2024-02-21T23:46:40.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.890+0000 7fb83ecf1b80 20 client.14742 trim_cache size 0 max 16384 2024-02-21T23:46:40.882 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-21T23:46:40.882 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "7ed81218-d112-11ee-95bd-87774f69a715", 2024-02-21T23:46:40.882 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 0, 2024-02-21T23:46:40.882 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 24500, 2024-02-21T23:46:40.882 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-02-21T23:46:40.882 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-02-21T23:46:40.883 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs", 2024-02-21T23:46:40.883 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 6.4455941809999997, 2024-02-21T23:46:40.883 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 17, 2024-02-21T23:46:40.883 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 85, 2024-02-21T23:46:40.883 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 84, 2024-02-21T23:46:40.883 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 11.875750803000001 2024-02-21T23:46:40.883 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-21T23:46:40.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.890+0000 7fb832b5e700 1 -- 172.21.15.90:0/1085379517 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fb81c0a1120 msgr2=0x7fb81c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:40.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.890+0000 7fb832b5e700 1 --2- 172.21.15.90:0/1085379517 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fb81c0a1120 0x7fb81c0a35e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb8180037f0 tx=0x7fb818005fb0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.890+0000 7fb832b5e700 1 -- 172.21.15.90:0/1085379517 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb82c0fefc0 msgr2=0x7fb82c196c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:40.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.890+0000 7fb832b5e700 1 --2- 172.21.15.90:0/1085379517 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb82c0fefc0 0x7fb82c196c20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb82000d980 tx=0x7fb82000dc90 comp rx=0 tx=0).stop 2024-02-21T23:46:40.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.891+0000 7fb832b5e700 1 -- 172.21.15.90:0/1085379517 shutdown_connections 2024-02-21T23:46:40.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.891+0000 7fb832b5e700 1 --2- 172.21.15.90:0/1085379517 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fb81c0a1120 0x7fb81c0a35e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.891+0000 7fb832b5e700 1 --2- 172.21.15.90:0/1085379517 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb82c102930 0x7fb82c19e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.891+0000 7fb832b5e700 1 --2- 172.21.15.90:0/1085379517 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb82c0fefc0 0x7fb82c196c20 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.891+0000 7fb832b5e700 1 --2- 172.21.15.90:0/1085379517 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb82c101f90 0x7fb82c197160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:40.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.891+0000 7fb832b5e700 1 -- 172.21.15.90:0/1085379517 >> 172.21.15.90:0/1085379517 conn(0x7fb82c0fa8f0 msgr2=0x7fb82c105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:40.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.891+0000 7fb832b5e700 1 -- 172.21.15.90:0/1085379517 shutdown_connections 2024-02-21T23:46:40.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:40.891+0000 7fb832b5e700 1 -- 172.21.15.90:0/1085379517 wait complete. 2024-02-21T23:46:40.887 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-02-21T23:46:40.887 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-02-21T23:46:40.888 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c1dc18f444603397768fcb7c34a8488b10284df6 shell --fsid 7ed81218-d112-11ee-95bd-87774f69a715 -- ceph fs ls 2024-02-21T23:46:40.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:40 smithi178 ceph-mon[47547]: pgmap v228: 129 pgs: 3 creating+peering, 28 unknown, 98 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 11 op/s 2024-02-21T23:46:40.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:40 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/725991941' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-21T23:46:40.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:40 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/202032530' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-21T23:46:40.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:40 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/1979871382' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-21T23:46:41.102 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/mon.a/config 2024-02-21T23:46:41.512 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:41 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/2635142482' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-21T23:46:41.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:41 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/2635142482' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-21T23:46:41.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:41 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/2635142482' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-21T23:46:42.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.402+0000 7f48b7910700 1 -- 172.21.15.90:0/1846174902 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48b0106e80 msgr2=0x7f48b0107260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:42.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.402+0000 7f48b7910700 1 --2- 172.21.15.90:0/1846174902 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48b0106e80 0x7f48b0107260 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f48a0009990 tx=0x7f48a0009ca0 comp rx=0 tx=0).stop 2024-02-21T23:46:42.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.402+0000 7f48b7910700 1 -- 172.21.15.90:0/1846174902 shutdown_connections 2024-02-21T23:46:42.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.402+0000 7f48b7910700 1 --2- 172.21.15.90:0/1846174902 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f48b0069100 0x7f48b010db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:42.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.402+0000 7f48b7910700 1 --2- 172.21.15.90:0/1846174902 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48b0106e80 0x7f48b0107260 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:42.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.402+0000 7f48b7910700 1 --2- 172.21.15.90:0/1846174902 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48b0068740 0x7f48b0068bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:42.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.402+0000 7f48b7910700 1 -- 172.21.15.90:0/1846174902 >> 172.21.15.90:0/1846174902 conn(0x7f48b0076d70 msgr2=0x7f48b0077180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:42.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.402+0000 7f48b7910700 1 -- 172.21.15.90:0/1846174902 shutdown_connections 2024-02-21T23:46:42.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.402+0000 7f48b7910700 1 -- 172.21.15.90:0/1846174902 wait complete. 2024-02-21T23:46:42.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.403+0000 7f48b7910700 1 Processor -- start 2024-02-21T23:46:42.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.403+0000 7f48b7910700 1 -- start start 2024-02-21T23:46:42.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.403+0000 7f48b7910700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48b0068740 0x7f48b0197100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:42.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.403+0000 7f48b7910700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48b0069100 0x7f48b0197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:42.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.403+0000 7f48b7910700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f48b0106e80 0x7f48b019b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:42.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.403+0000 7f48b7910700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b019c000 con 0x7f48b0069100 2024-02-21T23:46:42.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.403+0000 7f48b7910700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b019c170 con 0x7f48b0068740 2024-02-21T23:46:42.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.403+0000 7f48b7910700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b019c330 con 0x7f48b0106e80 2024-02-21T23:46:42.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.403+0000 7f48b4eab700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48b0069100 0x7f48b0197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:42.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.403+0000 7f48b4eab700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48b0069100 0x7f48b0197640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49324/0 (socket says 172.21.15.90:49324) 2024-02-21T23:46:42.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.403+0000 7f48b4eab700 1 -- 172.21.15.90:0/258690095 learned_addr learned my addr 172.21.15.90:0/258690095 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:42.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.403+0000 7f48b56ac700 1 --2- 172.21.15.90:0/258690095 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48b0068740 0x7f48b0197100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:42.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.403+0000 7f48b5ead700 1 --2- 172.21.15.90:0/258690095 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f48b0106e80 0x7f48b019b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:42.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.403+0000 7f48b4eab700 1 -- 172.21.15.90:0/258690095 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f48b0106e80 msgr2=0x7f48b019b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:42.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.403+0000 7f48b4eab700 1 --2- 172.21.15.90:0/258690095 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f48b0106e80 0x7f48b019b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:42.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.403+0000 7f48b4eab700 1 -- 172.21.15.90:0/258690095 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48b0068740 msgr2=0x7f48b0197100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:42.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.403+0000 7f48b4eab700 1 --2- 172.21.15.90:0/258690095 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48b0068740 0x7f48b0197100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:42.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.403+0000 7f48b4eab700 1 -- 172.21.15.90:0/258690095 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a00095f0 con 0x7f48b0069100 2024-02-21T23:46:42.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.403+0000 7f48b4eab700 1 --2- 172.21.15.90:0/258690095 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48b0069100 0x7f48b0197640 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f48ac00b530 tx=0x7f48ac00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:42.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.404+0000 7f48a67fc700 1 -- 172.21.15.90:0/258690095 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ac011830 con 0x7f48b0069100 2024-02-21T23:46:42.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.404+0000 7f48a67fc700 1 -- 172.21.15.90:0/258690095 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48ac011e50 con 0x7f48b0069100 2024-02-21T23:46:42.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.404+0000 7f48b7910700 1 -- 172.21.15.90:0/258690095 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48b019c610 con 0x7f48b0069100 2024-02-21T23:46:42.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.404+0000 7f48a67fc700 1 -- 172.21.15.90:0/258690095 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ac00f2e0 con 0x7f48b0069100 2024-02-21T23:46:42.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.404+0000 7f48b7910700 1 -- 172.21.15.90:0/258690095 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48b01a3e70 con 0x7f48b0069100 2024-02-21T23:46:42.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.404+0000 7f48b7910700 1 -- 172.21.15.90:0/258690095 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48b0109210 con 0x7f48b0069100 2024-02-21T23:46:42.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.406+0000 7f48a67fc700 1 -- 172.21.15.90:0/258690095 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7f48ac010580 con 0x7f48b0069100 2024-02-21T23:46:42.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.406+0000 7f48a67fc700 1 --2- 172.21.15.90:0/258690095 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f489c0a1120 0x7f489c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:42.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.406+0000 7f48b56ac700 1 --2- 172.21.15.90:0/258690095 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f489c0a1120 0x7f489c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:42.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.406+0000 7f48a67fc700 1 -- 172.21.15.90:0/258690095 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f48ac0ccba0 con 0x7f48b0069100 2024-02-21T23:46:42.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.406+0000 7f48b56ac700 1 --2- 172.21.15.90:0/258690095 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f489c0a1120 0x7f489c0a35e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f48a0006010 tx=0x7f48a000b540 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:42.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.410+0000 7f48a67fc700 1 -- 172.21.15.90:0/258690095 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f48ac0cce60 con 0x7f48b0069100 2024-02-21T23:46:42.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:42 smithi090 ceph-mon[43385]: pgmap v229: 129 pgs: 3 creating+peering, 11 unknown, 115 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 7 op/s 2024-02-21T23:46:42.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.648+0000 7f48b7910700 1 -- 172.21.15.90:0/258690095 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f48b0198410 con 0x7f48b0069100 2024-02-21T23:46:42.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.649+0000 7f48a67fc700 1 -- 172.21.15.90:0/258690095 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+88 (secure 0 0 0) 0x7f48ac015070 con 0x7f48b0069100 2024-02-21T23:46:42.634 INFO:teuthology.orchestra.run.smithi090.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-02-21T23:46:42.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.651+0000 7f48b7910700 1 -- 172.21.15.90:0/258690095 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f489c0a1120 msgr2=0x7f489c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:42.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.651+0000 7f48b7910700 1 --2- 172.21.15.90:0/258690095 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f489c0a1120 0x7f489c0a35e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f48a0006010 tx=0x7f48a000b540 comp rx=0 tx=0).stop 2024-02-21T23:46:42.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.651+0000 7f48b7910700 1 -- 172.21.15.90:0/258690095 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48b0069100 msgr2=0x7f48b0197640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:42.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.651+0000 7f48b7910700 1 --2- 172.21.15.90:0/258690095 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48b0069100 0x7f48b0197640 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f48ac00b530 tx=0x7f48ac00b8f0 comp rx=0 tx=0).stop 2024-02-21T23:46:42.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.651+0000 7f48b7910700 1 -- 172.21.15.90:0/258690095 shutdown_connections 2024-02-21T23:46:42.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.651+0000 7f48b7910700 1 --2- 172.21.15.90:0/258690095 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f489c0a1120 0x7f489c0a35e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:42.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.651+0000 7f48b7910700 1 --2- 172.21.15.90:0/258690095 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f48b0106e80 0x7f48b019b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:42.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.651+0000 7f48b7910700 1 --2- 172.21.15.90:0/258690095 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48b0068740 0x7f48b0197100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:42.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.651+0000 7f48b7910700 1 --2- 172.21.15.90:0/258690095 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48b0069100 0x7f48b0197640 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:42.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.651+0000 7f48b7910700 1 -- 172.21.15.90:0/258690095 >> 172.21.15.90:0/258690095 conn(0x7f48b0076d70 msgr2=0x7f48b00fe970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:42.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.651+0000 7f48b7910700 1 -- 172.21.15.90:0/258690095 shutdown_connections 2024-02-21T23:46:42.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:42.651+0000 7f48b7910700 1 -- 172.21.15.90:0/258690095 wait complete. 2024-02-21T23:46:42.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:42 smithi176 ceph-mon[47172]: pgmap v229: 129 pgs: 3 creating+peering, 11 unknown, 115 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 7 op/s 2024-02-21T23:46:42.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:42 smithi178 ceph-mon[47547]: pgmap v229: 129 pgs: 3 creating+peering, 11 unknown, 115 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 7 op/s 2024-02-21T23:46:43.006 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-02-21T23:46:43.007 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-21T23:46:43.007 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi090.front.sepia.ceph.com 2024-02-21T23:46:43.007 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-02-21T23:46:43.007 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-02-21T23:46:43.007 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-02-21T23:46:43.007 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-02-21T23:46:43.007 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-21T23:46:43.007 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-02-21T23:46:43.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:46:43.007 DEBUG:teuthology.orchestra.run.smithi090:> ip addr 2024-02-21T23:46:43.028 INFO:teuthology.orchestra.run.smithi090.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-21T23:46:43.028 INFO:teuthology.orchestra.run.smithi090.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-21T23:46:43.028 INFO:teuthology.orchestra.run.smithi090.stdout: inet 127.0.0.1/8 scope host lo 2024-02-21T23:46:43.028 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-02-21T23:46:43.028 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 ::1/128 scope host 2024-02-21T23:46:43.029 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-02-21T23:46:43.029 INFO:teuthology.orchestra.run.smithi090.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-21T23:46:43.029 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a8 brd ff:ff:ff:ff:ff:ff 2024-02-21T23:46:43.029 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f0 2024-02-21T23:46:43.029 INFO:teuthology.orchestra.run.smithi090.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-21T23:46:43.029 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a9 brd ff:ff:ff:ff:ff:ff 2024-02-21T23:46:43.029 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f1 2024-02-21T23:46:43.029 INFO:teuthology.orchestra.run.smithi090.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-21T23:46:43.029 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:66 brd ff:ff:ff:ff:ff:ff 2024-02-21T23:46:43.029 INFO:teuthology.orchestra.run.smithi090.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-21T23:46:43.030 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:67 brd ff:ff:ff:ff:ff:ff 2024-02-21T23:46:43.030 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.21.15.90/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-02-21T23:46:43.030 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft 41909sec preferred_lft 41909sec 2024-02-21T23:46:43.030 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::b06a:4dab:9131:21e7/64 scope link noprefixroute 2024-02-21T23:46:43.030 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-02-21T23:46:43.030 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-02-21T23:46:43.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:46:43.031 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2024-02-21T23:46:43.031 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-02-21T23:46:43.031 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link add name ceph-brx type bridge 2024-02-21T23:46:43.031 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip addr flush dev ceph-brx 2024-02-21T23:46:43.031 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set ceph-brx up 2024-02-21T23:46:43.031 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-02-21T23:46:43.031 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-02-21T23:46:43.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:46:43.242 DEBUG:teuthology.orchestra.run.smithi090:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-21T23:46:43.271 INFO:teuthology.orchestra.run.smithi090.stdout:1 2024-02-21T23:46:43.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:46:43.272 DEBUG:teuthology.orchestra.run.smithi090:> route 2024-02-21T23:46:43.343 INFO:teuthology.orchestra.run.smithi090.stdout:Kernel IP routing table 2024-02-21T23:46:43.343 INFO:teuthology.orchestra.run.smithi090.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-21T23:46:43.343 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-02-21T23:46:43.343 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-02-21T23:46:43.343 INFO:teuthology.orchestra.run.smithi090.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-21T23:46:43.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:46:43.343 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2024-02-21T23:46:43.344 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-02-21T23:46:43.344 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-02-21T23:46:43.344 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-02-21T23:46:43.344 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-02-21T23:46:43.344 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-02-21T23:46:43.487 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:43 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/258690095' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-21T23:46:43.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:46:43.578 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-02-21T23:46:43.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:46:43.596 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list-id 2024-02-21T23:46:43.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:46:43.653 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2024-02-21T23:46:43.654 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-02-21T23:46:43.654 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-02-21T23:46:43.654 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-02-21T23:46:43.654 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-02-21T23:46:43.749 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-02-21T23:46:43.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:46:43.749 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2024-02-21T23:46:43.749 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-02-21T23:46:43.749 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-02-21T23:46:43.750 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-02-21T23:46:43.750 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-02-21T23:46:43.750 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-02-21T23:46:43.750 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-02-21T23:46:43.750 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-02-21T23:46:43.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:43 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/258690095' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-21T23:46:43.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:43 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/258690095' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-21T23:46:43.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:46:43.975 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2024-02-21T23:46:43.975 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-02-21T23:46:43.975 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set brx.0 up 2024-02-21T23:46:43.975 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set dev brx.0 master ceph-brx 2024-02-21T23:46:43.975 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-02-21T23:46:44.073 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-02-21T23:46:44.073 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-21T23:46:44.073 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-02-21T23:46:44.090 INFO:teuthology.orchestra.run.smithi090.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-02-21T23:46:44.090 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-21T23:46:44.090 DEBUG:teuthology.orchestra.run.smithi090:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-02-21T23:46:44.146 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-02-21T23:46:44.214 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-02-21T23:46:44.214 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-02-21T23:46:44.282 INFO:teuthology.orchestra.run.smithi090.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-02-21T23:46:44.283 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-02-21T23:46:44.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:46:44.284 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-02-21T23:46:44.342 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-02-21T23:46:44.342 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-02-21T23:46:44.384 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-02-21T23:46:44.410 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-02-21T23:46:44.411 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-02-21T23:46:44.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi090.stderr:2024-02-21T23:46:44.489+0000 7f8c7acee5c0 -1 init, newargv = 0x55dceb741570 newargc=15 2024-02-21T23:46:44.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi090.stderr:ceph-fuse[97587]: starting ceph client 2024-02-21T23:46:44.478 INFO:teuthology.orchestra.run.smithi090.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-02-21T23:46:44.480 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-02-21T23:46:44.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:46:44.480 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-02-21T23:46:44.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi090.stderr:ceph-fuse[97587]: starting fuse 2024-02-21T23:46:44.497 INFO:teuthology.orchestra.run.smithi090.stdout:124 2024-02-21T23:46:44.498 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [124] 2024-02-21T23:46:44.498 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-02-21T23:46:44.498 DEBUG:teuthology.orchestra.run.smithi090:> import glob 2024-02-21T23:46:44.498 DEBUG:teuthology.orchestra.run.smithi090:> import re 2024-02-21T23:46:44.498 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-02-21T23:46:44.498 DEBUG:teuthology.orchestra.run.smithi090:> import subprocess 2024-02-21T23:46:44.498 DEBUG:teuthology.orchestra.run.smithi090:> 2024-02-21T23:46:44.498 DEBUG:teuthology.orchestra.run.smithi090:> def _find_admin_socket(client_name): 2024-02-21T23:46:44.498 DEBUG:teuthology.orchestra.run.smithi090:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-02-21T23:46:44.498 DEBUG:teuthology.orchestra.run.smithi090:> files = glob.glob(asok_path) 2024-02-21T23:46:44.498 DEBUG:teuthology.orchestra.run.smithi090:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-02-21T23:46:44.499 DEBUG:teuthology.orchestra.run.smithi090:> 2024-02-21T23:46:44.499 DEBUG:teuthology.orchestra.run.smithi090:> # Given a non-glob path, it better be there 2024-02-21T23:46:44.499 DEBUG:teuthology.orchestra.run.smithi090:> if "*" not in asok_path: 2024-02-21T23:46:44.499 DEBUG:teuthology.orchestra.run.smithi090:> assert(len(files) == 1) 2024-02-21T23:46:44.499 DEBUG:teuthology.orchestra.run.smithi090:> return files[0] 2024-02-21T23:46:44.499 DEBUG:teuthology.orchestra.run.smithi090:> 2024-02-21T23:46:44.499 DEBUG:teuthology.orchestra.run.smithi090:> for f in files: 2024-02-21T23:46:44.499 DEBUG:teuthology.orchestra.run.smithi090:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-02-21T23:46:44.499 DEBUG:teuthology.orchestra.run.smithi090:> if os.path.exists("/proc/{0}".format(pid)): 2024-02-21T23:46:44.499 DEBUG:teuthology.orchestra.run.smithi090:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-02-21T23:46:44.499 DEBUG:teuthology.orchestra.run.smithi090:> contents = proc_f.read() 2024-02-21T23:46:44.499 DEBUG:teuthology.orchestra.run.smithi090:> if mountpoint in contents: 2024-02-21T23:46:44.499 DEBUG:teuthology.orchestra.run.smithi090:> return f 2024-02-21T23:46:44.499 DEBUG:teuthology.orchestra.run.smithi090:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-02-21T23:46:44.499 DEBUG:teuthology.orchestra.run.smithi090:> 2024-02-21T23:46:44.500 DEBUG:teuthology.orchestra.run.smithi090:> print(_find_admin_socket("client.admin")) 2024-02-21T23:46:44.500 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-02-21T23:46:44.561 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:44 smithi090 ceph-mon[43385]: pgmap v230: 129 pgs: 1 creating+peering, 128 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-02-21T23:46:44.621 INFO:teuthology.orchestra.run.smithi090.stdout:/var/run/ceph/ceph-client.admin.97587.asok 2024-02-21T23:46:44.804 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.97587.asok 2024-02-21T23:46:44.804 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:46:44.804 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.97587.asok status 2024-02-21T23:46:44.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:44 smithi176 ceph-mon[47172]: pgmap v230: 129 pgs: 1 creating+peering, 128 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-02-21T23:46:44.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:44 smithi178 ceph-mon[47547]: pgmap v230: 129 pgs: 1 creating+peering, 128 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-02-21T23:46:44.960 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-21T23:46:44.960 INFO:teuthology.orchestra.run.smithi090.stdout: "metadata": { 2024-02-21T23:46:44.960 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_sha1": "c1dc18f444603397768fcb7c34a8488b10284df6", 2024-02-21T23:46:44.960 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_version": "ceph version 18.2.1-638-gc1dc18f4 (c1dc18f444603397768fcb7c34a8488b10284df6) reef (stable)", 2024-02-21T23:46:44.961 INFO:teuthology.orchestra.run.smithi090.stdout: "entity_id": "admin", 2024-02-21T23:46:44.961 INFO:teuthology.orchestra.run.smithi090.stdout: "hostname": "smithi090", 2024-02-21T23:46:44.961 INFO:teuthology.orchestra.run.smithi090.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-02-21T23:46:44.961 INFO:teuthology.orchestra.run.smithi090.stdout: "pid": "97587", 2024-02-21T23:46:44.961 INFO:teuthology.orchestra.run.smithi090.stdout: "root": "/" 2024-02-21T23:46:44.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-21T23:46:44.961 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_count": 0, 2024-02-21T23:46:44.961 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_pinned_count": 0, 2024-02-21T23:46:44.961 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 14754, 2024-02-21T23:46:44.961 INFO:teuthology.orchestra.run.smithi090.stdout: "inst": { 2024-02-21T23:46:44.961 INFO:teuthology.orchestra.run.smithi090.stdout: "name": { 2024-02-21T23:46:44.961 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "client", 2024-02-21T23:46:44.961 INFO:teuthology.orchestra.run.smithi090.stdout: "num": 14754 2024-02-21T23:46:44.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-21T23:46:44.961 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-02-21T23:46:44.962 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-02-21T23:46:44.962 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-02-21T23:46:44.962 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 6383149 2024-02-21T23:46:44.962 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-21T23:46:44.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-21T23:46:44.962 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-02-21T23:46:44.962 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-02-21T23:46:44.962 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-02-21T23:46:44.962 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 6383149 2024-02-21T23:46:44.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-21T23:46:44.962 INFO:teuthology.orchestra.run.smithi090.stdout: "inst_str": "client.14754 192.168.0.1:0/6383149", 2024-02-21T23:46:44.962 INFO:teuthology.orchestra.run.smithi090.stdout: "addr_str": "192.168.0.1:0/6383149", 2024-02-21T23:46:44.962 INFO:teuthology.orchestra.run.smithi090.stdout: "inode_count": 1, 2024-02-21T23:46:44.962 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_epoch": 17, 2024-02-21T23:46:44.963 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch": 85, 2024-02-21T23:46:44.963 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch_barrier": 0, 2024-02-21T23:46:44.963 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklisted": false, 2024-02-21T23:46:44.963 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs" 2024-02-21T23:46:44.963 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-21T23:46:44.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:46:44.971 DEBUG:teuthology.orchestra.run.smithi090:> stat --file-system '--printf=%T 2024-02-21T23:46:44.971 DEBUG:teuthology.orchestra.run.smithi090:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-02-21T23:46:44.988 INFO:teuthology.orchestra.run.smithi090.stdout:fuseblk 2024-02-21T23:46:44.988 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-02-21T23:46:44.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:46:44.989 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-02-21T23:46:45.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:46:45.057 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-02-21T23:46:45.167 INFO:teuthology.orchestra.run.smithi090.stdout:# file: . 2024-02-21T23:46:45.167 INFO:teuthology.orchestra.run.smithi090.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-02-21T23:46:45.167 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:46:45.168 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-21T23:46:45.168 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:46:45.186 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:46:45.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.363+0000 7f4e4347f700 1 -- 172.21.15.90:0/4086818599 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e3c1016a0 msgr2=0x7f4e3c107f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:45.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.363+0000 7f4e4347f700 1 --2- 172.21.15.90:0/4086818599 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e3c1016a0 0x7f4e3c107f50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4e3400b0d0 tx=0x7f4e3400b3e0 comp rx=0 tx=0).stop 2024-02-21T23:46:45.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.363+0000 7f4e4347f700 1 -- 172.21.15.90:0/4086818599 shutdown_connections 2024-02-21T23:46:45.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.363+0000 7f4e4347f700 1 --2- 172.21.15.90:0/4086818599 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e3c1016a0 0x7f4e3c107f50 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:45.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.363+0000 7f4e4347f700 1 --2- 172.21.15.90:0/4086818599 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4e3c0ffb10 0x7f4e3c0fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:45.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.363+0000 7f4e4347f700 1 --2- 172.21.15.90:0/4086818599 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e3c100d00 0x7f4e3c101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:45.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.363+0000 7f4e4347f700 1 -- 172.21.15.90:0/4086818599 >> 172.21.15.90:0/4086818599 conn(0x7f4e3c0fb080 msgr2=0x7f4e3c0fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:45.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.363+0000 7f4e4347f700 1 -- 172.21.15.90:0/4086818599 shutdown_connections 2024-02-21T23:46:45.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.363+0000 7f4e4347f700 1 -- 172.21.15.90:0/4086818599 wait complete. 2024-02-21T23:46:45.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.364+0000 7f4e4347f700 1 Processor -- start 2024-02-21T23:46:45.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.364+0000 7f4e4347f700 1 -- start start 2024-02-21T23:46:45.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.364+0000 7f4e4347f700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4e3c0ffb10 0x7f4e3c069b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:45.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.364+0000 7f4e4347f700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e3c100d00 0x7f4e3c19b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:45.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.364+0000 7f4e4347f700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e3c1016a0 0x7f4e3c19dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:45.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.364+0000 7f4e4347f700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3c19e770 con 0x7f4e3c100d00 2024-02-21T23:46:45.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.364+0000 7f4e4347f700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3c19e8e0 con 0x7f4e3c1016a0 2024-02-21T23:46:45.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.364+0000 7f4e4347f700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3c19eaa0 con 0x7f4e3c0ffb10 2024-02-21T23:46:45.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.364+0000 7f4e40a1a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e3c100d00 0x7f4e3c19b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:45.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.364+0000 7f4e40a1a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e3c100d00 0x7f4e3c19b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49344/0 (socket says 172.21.15.90:49344) 2024-02-21T23:46:45.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.364+0000 7f4e40a1a700 1 -- 172.21.15.90:0/2540012388 learned_addr learned my addr 172.21.15.90:0/2540012388 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:45.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.364+0000 7f4e4121b700 1 --2- 172.21.15.90:0/2540012388 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4e3c0ffb10 0x7f4e3c069b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:45.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.364+0000 7f4e41a1c700 1 --2- 172.21.15.90:0/2540012388 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e3c1016a0 0x7f4e3c19dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:45.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.364+0000 7f4e40a1a700 1 -- 172.21.15.90:0/2540012388 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4e3c0ffb10 msgr2=0x7f4e3c069b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:45.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.364+0000 7f4e40a1a700 1 --2- 172.21.15.90:0/2540012388 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4e3c0ffb10 0x7f4e3c069b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:45.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.364+0000 7f4e40a1a700 1 -- 172.21.15.90:0/2540012388 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e3c1016a0 msgr2=0x7f4e3c19dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:45.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.364+0000 7f4e40a1a700 1 --2- 172.21.15.90:0/2540012388 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e3c1016a0 0x7f4e3c19dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:45.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.365+0000 7f4e40a1a700 1 -- 172.21.15.90:0/2540012388 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e34009d00 con 0x7f4e3c100d00 2024-02-21T23:46:45.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.365+0000 7f4e4121b700 1 --2- 172.21.15.90:0/2540012388 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4e3c0ffb10 0x7f4e3c069b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:46:45.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.365+0000 7f4e40a1a700 1 --2- 172.21.15.90:0/2540012388 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e3c100d00 0x7f4e3c19b9e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4e3000d950 tx=0x7f4e3000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:45.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.365+0000 7f4e2e7fc700 1 -- 172.21.15.90:0/2540012388 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e30009b40 con 0x7f4e3c100d00 2024-02-21T23:46:45.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.365+0000 7f4e2e7fc700 1 -- 172.21.15.90:0/2540012388 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e30009ca0 con 0x7f4e3c100d00 2024-02-21T23:46:45.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.365+0000 7f4e4347f700 1 -- 172.21.15.90:0/2540012388 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e3c19ed80 con 0x7f4e3c100d00 2024-02-21T23:46:45.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.365+0000 7f4e2e7fc700 1 -- 172.21.15.90:0/2540012388 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3000f620 con 0x7f4e3c100d00 2024-02-21T23:46:45.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.365+0000 7f4e4347f700 1 -- 172.21.15.90:0/2540012388 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e3c19f230 con 0x7f4e3c100d00 2024-02-21T23:46:45.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.366+0000 7f4e4347f700 1 -- 172.21.15.90:0/2540012388 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e3c106f40 con 0x7f4e3c100d00 2024-02-21T23:46:45.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.367+0000 7f4e2e7fc700 1 -- 172.21.15.90:0/2540012388 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7f4e30010460 con 0x7f4e3c100d00 2024-02-21T23:46:45.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.367+0000 7f4e2e7fc700 1 --2- 172.21.15.90:0/2540012388 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f4e280a1120 0x7f4e280a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:45.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.368+0000 7f4e4121b700 1 --2- 172.21.15.90:0/2540012388 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f4e280a1120 0x7f4e280a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:45.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.368+0000 7f4e2e7fc700 1 -- 172.21.15.90:0/2540012388 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f4e300cce70 con 0x7f4e3c100d00 2024-02-21T23:46:45.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.368+0000 7f4e4121b700 1 --2- 172.21.15.90:0/2540012388 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f4e280a1120 0x7f4e280a35e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c100b60 tx=0x7f4e38009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:45.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.371+0000 7f4e2e7fc700 1 -- 172.21.15.90:0/2540012388 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f4e30096410 con 0x7f4e3c100d00 2024-02-21T23:46:45.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.577+0000 7f4e4347f700 1 -- 172.21.15.90:0/2540012388 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e3c067110 con 0x7f4e3c100d00 2024-02-21T23:46:45.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.578+0000 7f4e2e7fc700 1 -- 172.21.15.90:0/2540012388 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23925 (secure 0 0 0) 0x7f4e30020090 con 0x7f4e3c100d00 2024-02-21T23:46:45.564 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:46:45.564 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":85,"fsid":"7ed81218-d112-11ee-95bd-87774f69a715","created":"2024-02-21T23:40:41.785526+0000","modified":"2024-02-21T23:46:38.878698+0000","last_up_change":"2024-02-21T23:46:00.860501+0000","last_in_change":"2024-02-21T23:45:44.423815+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:43:21.675789+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:46:32.337047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-21T23:46:33.622440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-21T23:46:36.686123+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"77878890-d9a5-4d5b-a943-fec29ee0446e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6803","nonce":885319826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6805","nonce":885319826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6809","nonce":885319826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6807","nonce":885319826}]},"public_addr":"172.21.15.90:6803/885319826","cluster_addr":"172.21.15.90:6805/885319826","heartbeat_back_addr":"172.21.15.90:6809/885319826","heartbeat_front_addr":"172.21.15.90:6807/885319826","state":["exists","up"]},{"osd":1,"uuid":"7d6ea950-7a57-4f13-ab38-10a658912783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6801","nonce":4085240925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6803","nonce":4085240925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6807","nonce":4085240925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6805","nonce":4085240925}]},"public_addr":"172.21.15.176:6801/4085240925","cluster_addr":"172.21.15.176:6803/4085240925","heartbeat_back_addr":"172.21.15.176:6807/4085240925","heartbeat_front_addr":"172.21.15.176:6805/4085240925","state":["exists","up"]},{"osd":2,"uuid":"b83486bb-27f2-4d56-97fb-fb336fec4b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6801","nonce":3597250789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6803","nonce":3597250789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6807","nonce":3597250789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6805","nonce":3597250789}]},"public_addr":"172.21.15.178:6801/3597250789","cluster_addr":"172.21.15.178:6803/3597250789","heartbeat_back_addr":"172.21.15.178:6807/3597250789","heartbeat_front_addr":"172.21.15.178:6805/3597250789","state":["exists","up"]},{"osd":3,"uuid":"870e5bb6-eed5-4a0a-8e03-38de4f816b44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6811","nonce":3299043821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6813","nonce":3299043821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6817","nonce":3299043821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6815","nonce":3299043821}]},"public_addr":"172.21.15.90:6811/3299043821","cluster_addr":"172.21.15.90:6813/3299043821","heartbeat_back_addr":"172.21.15.90:6817/3299043821","heartbeat_front_addr":"172.21.15.90:6815/3299043821","state":["exists","up"]},{"osd":4,"uuid":"cb9f7f48-76e0-4368-ab87-2a09a8487ccd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6809","nonce":3548915687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6811","nonce":3548915687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6815","nonce":3548915687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6813","nonce":3548915687}]},"public_addr":"172.21.15.176:6809/3548915687","cluster_addr":"172.21.15.176:6811/3548915687","heartbeat_back_addr":"172.21.15.176:6815/3548915687","heartbeat_front_addr":"172.21.15.176:6813/3548915687","state":["exists","up"]},{"osd":5,"uuid":"54749022-c50d-4831-a1d6-8412b59c8a01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6809","nonce":286274518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6811","nonce":286274518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6815","nonce":286274518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6813","nonce":286274518}]},"public_addr":"172.21.15.178:6809/286274518","cluster_addr":"172.21.15.178:6811/286274518","heartbeat_back_addr":"172.21.15.178:6815/286274518","heartbeat_front_addr":"172.21.15.178:6813/286274518","state":["exists","up"]},{"osd":6,"uuid":"09bb5393-2efd-4a63-b0e8-99bef5bced13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6819","nonce":67640620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6821","nonce":67640620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6825","nonce":67640620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6823","nonce":67640620}]},"public_addr":"172.21.15.90:6819/67640620","cluster_addr":"172.21.15.90:6821/67640620","heartbeat_back_addr":"172.21.15.90:6825/67640620","heartbeat_front_addr":"172.21.15.90:6823/67640620","state":["exists","up"]},{"osd":7,"uuid":"7a859e69-6b63-467b-a3b3-99247d66f5c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6817","nonce":3998561598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6819","nonce":3998561598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6823","nonce":3998561598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6821","nonce":3998561598}]},"public_addr":"172.21.15.176:6817/3998561598","cluster_addr":"172.21.15.176:6819/3998561598","heartbeat_back_addr":"172.21.15.176:6823/3998561598","heartbeat_front_addr":"172.21.15.176:6821/3998561598","state":["exists","up"]},{"osd":8,"uuid":"bc23144e-22a5-4894-a8b5-9323383574d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6817","nonce":490229963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6819","nonce":490229963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6823","nonce":490229963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6821","nonce":490229963}]},"public_addr":"172.21.15.178:6817/490229963","cluster_addr":"172.21.15.178:6819/490229963","heartbeat_back_addr":"172.21.15.178:6823/490229963","heartbeat_front_addr":"172.21.15.178:6821/490229963","state":["exists","up"]},{"osd":9,"uuid":"0c2cecd4-55de-4fc4-a6f4-ca97c8460444","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6827","nonce":2061394761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6829","nonce":2061394761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6833","nonce":2061394761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6831","nonce":2061394761}]},"public_addr":"172.21.15.90:6827/2061394761","cluster_addr":"172.21.15.90:6829/2061394761","heartbeat_back_addr":"172.21.15.90:6833/2061394761","heartbeat_front_addr":"172.21.15.90:6831/2061394761","state":["exists","up"]},{"osd":10,"uuid":"fa04232b-b1ec-447a-8e94-7c9bf26d0291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6825","nonce":639411757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6827","nonce":639411757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6831","nonce":639411757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6829","nonce":639411757}]},"public_addr":"172.21.15.176:6825/639411757","cluster_addr":"172.21.15.176:6827/639411757","heartbeat_back_addr":"172.21.15.176:6831/639411757","heartbeat_front_addr":"172.21.15.176:6829/639411757","state":["exists","up"]},{"osd":11,"uuid":"7282e294-64fc-4856-9c74-86115f69408c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6825","nonce":4144536256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6827","nonce":4144536256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6831","nonce":4144536256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6829","nonce":4144536256}]},"public_addr":"172.21.15.178:6825/4144536256","cluster_addr":"172.21.15.178:6827/4144536256","heartbeat_back_addr":"172.21.15.178:6831/4144536256","heartbeat_front_addr":"172.21.15.178:6829/4144536256","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:42:44.280345+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:01.235630+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:19.408896+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:36.201749+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:53.401854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:44:11.342158+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:44:29.324502+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:44:46.821595+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:04.642344+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:23.019004+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:41.140149+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:59.401951+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/159655312":"2024-02-22T23:41:33.360021+0000","172.21.15.90:6801/1385551604":"2024-02-22T23:41:33.360021+0000","172.21.15.90:0/1452574770":"2024-02-22T23:41:33.360021+0000","172.21.15.90:0/3815292343":"2024-02-22T23:41:11.484145+0000","172.21.15.90:6800/1385551604":"2024-02-22T23:41:33.360021+0000","172.21.15.90:6801/4247695574":"2024-02-22T23:41:11.484145+0000","172.21.15.90:0/1328266622":"2024-02-22T23:41:11.484145+0000","172.21.15.90:6800/4247695574":"2024-02-22T23:41:11.484145+0000","172.21.15.90:0/2569421522":"2024-02-22T23:41:11.484145+0000","172.21.15.90:0/3019701027":"2024-02-22T23:41:33.360021+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:46:45.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.581+0000 7f4e4347f700 1 -- 172.21.15.90:0/2540012388 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f4e280a1120 msgr2=0x7f4e280a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:45.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.581+0000 7f4e4347f700 1 --2- 172.21.15.90:0/2540012388 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f4e280a1120 0x7f4e280a35e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c100b60 tx=0x7f4e38009260 comp rx=0 tx=0).stop 2024-02-21T23:46:45.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.581+0000 7f4e4347f700 1 -- 172.21.15.90:0/2540012388 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e3c100d00 msgr2=0x7f4e3c19b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:45.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.581+0000 7f4e4347f700 1 --2- 172.21.15.90:0/2540012388 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e3c100d00 0x7f4e3c19b9e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4e3000d950 tx=0x7f4e3000dd10 comp rx=0 tx=0).stop 2024-02-21T23:46:45.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.581+0000 7f4e4347f700 1 -- 172.21.15.90:0/2540012388 shutdown_connections 2024-02-21T23:46:45.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.581+0000 7f4e4347f700 1 --2- 172.21.15.90:0/2540012388 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f4e280a1120 0x7f4e280a35e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:45.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.581+0000 7f4e4347f700 1 --2- 172.21.15.90:0/2540012388 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e3c1016a0 0x7f4e3c19dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:45.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.581+0000 7f4e4347f700 1 --2- 172.21.15.90:0/2540012388 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4e3c0ffb10 0x7f4e3c069b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:45.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.581+0000 7f4e4347f700 1 --2- 172.21.15.90:0/2540012388 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e3c100d00 0x7f4e3c19b9e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:45.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.581+0000 7f4e4347f700 1 -- 172.21.15.90:0/2540012388 >> 172.21.15.90:0/2540012388 conn(0x7f4e3c0fb080 msgr2=0x7f4e3c104080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:45.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.581+0000 7f4e4347f700 1 -- 172.21.15.90:0/2540012388 shutdown_connections 2024-02-21T23:46:45.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.581+0000 7f4e4347f700 1 -- 172.21.15.90:0/2540012388 wait complete. 2024-02-21T23:46:45.578 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:43:21.675789+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-21T23:46:32.337047+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-21T23:46:33.622440+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-02-21T23:46:36.686123+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-02-21T23:46:45.579 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:46:45.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.719+0000 7efc63b29700 1 -- 172.21.15.90:0/3874380516 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc5c101f00 msgr2=0x7efc5c102360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:45.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.719+0000 7efc63b29700 1 --2- 172.21.15.90:0/3874380516 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc5c101f00 0x7efc5c102360 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7efc580099c0 tx=0x7efc58009cd0 comp rx=0 tx=0).stop 2024-02-21T23:46:45.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.719+0000 7efc63b29700 1 -- 172.21.15.90:0/3874380516 shutdown_connections 2024-02-21T23:46:45.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.719+0000 7efc63b29700 1 --2- 172.21.15.90:0/3874380516 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efc5c1028a0 0x7efc5c109150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:45.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.719+0000 7efc63b29700 1 --2- 172.21.15.90:0/3874380516 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc5c100d10 0x7efc5c101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:45.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.719+0000 7efc63b29700 1 --2- 172.21.15.90:0/3874380516 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc5c101f00 0x7efc5c102360 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:45.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.719+0000 7efc63b29700 1 -- 172.21.15.90:0/3874380516 >> 172.21.15.90:0/3874380516 conn(0x7efc5c0fc280 msgr2=0x7efc5c0fe6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:45.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.719+0000 7efc63b29700 1 -- 172.21.15.90:0/3874380516 shutdown_connections 2024-02-21T23:46:45.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.719+0000 7efc63b29700 1 -- 172.21.15.90:0/3874380516 wait complete. 2024-02-21T23:46:45.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.720+0000 7efc63b29700 1 Processor -- start 2024-02-21T23:46:45.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.720+0000 7efc63b29700 1 -- start start 2024-02-21T23:46:45.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.720+0000 7efc63b29700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efc5c100d10 0x7efc5c074ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:45.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.720+0000 7efc63b29700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc5c101f00 0x7efc5c073160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:45.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.720+0000 7efc63b29700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc5c1028a0 0x7efc5c0736a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:45.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.720+0000 7efc63b29700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc5c073e40 con 0x7efc5c101f00 2024-02-21T23:46:45.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.720+0000 7efc63b29700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc5c073fb0 con 0x7efc5c1028a0 2024-02-21T23:46:45.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.720+0000 7efc63b29700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc5c1a0cb0 con 0x7efc5c100d10 2024-02-21T23:46:45.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.720+0000 7efc610c4700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc5c101f00 0x7efc5c073160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:45.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.720+0000 7efc610c4700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc5c101f00 0x7efc5c073160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49358/0 (socket says 172.21.15.90:49358) 2024-02-21T23:46:45.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.720+0000 7efc610c4700 1 -- 172.21.15.90:0/1276792806 learned_addr learned my addr 172.21.15.90:0/1276792806 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:45.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.720+0000 7efc620c6700 1 --2- 172.21.15.90:0/1276792806 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc5c1028a0 0x7efc5c0736a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:45.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.720+0000 7efc618c5700 1 --2- 172.21.15.90:0/1276792806 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efc5c100d10 0x7efc5c074ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:45.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.720+0000 7efc610c4700 1 -- 172.21.15.90:0/1276792806 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efc5c100d10 msgr2=0x7efc5c074ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:45.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.720+0000 7efc610c4700 1 --2- 172.21.15.90:0/1276792806 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efc5c100d10 0x7efc5c074ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:45.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.720+0000 7efc610c4700 1 -- 172.21.15.90:0/1276792806 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc5c1028a0 msgr2=0x7efc5c0736a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:45.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.720+0000 7efc610c4700 1 --2- 172.21.15.90:0/1276792806 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc5c1028a0 0x7efc5c0736a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:45.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.720+0000 7efc610c4700 1 -- 172.21.15.90:0/1276792806 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc580095f0 con 0x7efc5c101f00 2024-02-21T23:46:45.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.720+0000 7efc620c6700 1 --2- 172.21.15.90:0/1276792806 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc5c1028a0 0x7efc5c0736a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:46:45.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.721+0000 7efc610c4700 1 --2- 172.21.15.90:0/1276792806 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc5c101f00 0x7efc5c073160 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7efc580036f0 tx=0x7efc580047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:45.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.721+0000 7efc52ffd700 1 -- 172.21.15.90:0/1276792806 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc5801d070 con 0x7efc5c101f00 2024-02-21T23:46:45.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.721+0000 7efc52ffd700 1 -- 172.21.15.90:0/1276792806 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efc58004110 con 0x7efc5c101f00 2024-02-21T23:46:45.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.721+0000 7efc63b29700 1 -- 172.21.15.90:0/1276792806 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc5c1a0ee0 con 0x7efc5c101f00 2024-02-21T23:46:45.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.721+0000 7efc52ffd700 1 -- 172.21.15.90:0/1276792806 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc58022e10 con 0x7efc5c101f00 2024-02-21T23:46:45.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.721+0000 7efc63b29700 1 -- 172.21.15.90:0/1276792806 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc5c1a12a0 con 0x7efc5c101f00 2024-02-21T23:46:45.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.722+0000 7efc63b29700 1 -- 172.21.15.90:0/1276792806 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc5c1081e0 con 0x7efc5c101f00 2024-02-21T23:46:45.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.723+0000 7efc52ffd700 1 -- 172.21.15.90:0/1276792806 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7efc58022470 con 0x7efc5c101f00 2024-02-21T23:46:45.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.723+0000 7efc52ffd700 1 --2- 172.21.15.90:0/1276792806 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7efc480a1120 0x7efc480a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:45.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.724+0000 7efc618c5700 1 --2- 172.21.15.90:0/1276792806 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7efc480a1120 0x7efc480a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:45.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.724+0000 7efc52ffd700 1 -- 172.21.15.90:0/1276792806 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7efc580ce010 con 0x7efc5c101f00 2024-02-21T23:46:45.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.724+0000 7efc618c5700 1 --2- 172.21.15.90:0/1276792806 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7efc480a1120 0x7efc480a35e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7efc4c005950 tx=0x7efc4c0058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:45.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.727+0000 7efc52ffd700 1 -- 172.21.15.90:0/1276792806 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7efc58097640 con 0x7efc5c101f00 2024-02-21T23:46:45.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.921+0000 7efc63b29700 1 -- 172.21.15.90:0/1276792806 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efc5c02d0f0 con 0x7efc5c101f00 2024-02-21T23:46:45.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.921+0000 7efc52ffd700 1 -- 172.21.15.90:0/1276792806 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7efc5c02d0f0 con 0x7efc5c101f00 2024-02-21T23:46:45.907 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-02-21T23:46:45.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.923+0000 7efc63b29700 1 -- 172.21.15.90:0/1276792806 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7efc480a1120 msgr2=0x7efc480a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:45.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.923+0000 7efc63b29700 1 --2- 172.21.15.90:0/1276792806 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7efc480a1120 0x7efc480a35e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7efc4c005950 tx=0x7efc4c0058e0 comp rx=0 tx=0).stop 2024-02-21T23:46:45.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.923+0000 7efc63b29700 1 -- 172.21.15.90:0/1276792806 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc5c101f00 msgr2=0x7efc5c073160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:45.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.923+0000 7efc63b29700 1 --2- 172.21.15.90:0/1276792806 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc5c101f00 0x7efc5c073160 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7efc580036f0 tx=0x7efc580047b0 comp rx=0 tx=0).stop 2024-02-21T23:46:45.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.923+0000 7efc63b29700 1 -- 172.21.15.90:0/1276792806 shutdown_connections 2024-02-21T23:46:45.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.923+0000 7efc63b29700 1 --2- 172.21.15.90:0/1276792806 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7efc480a1120 0x7efc480a35e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:45.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.923+0000 7efc63b29700 1 --2- 172.21.15.90:0/1276792806 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc5c1028a0 0x7efc5c0736a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:45.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.923+0000 7efc63b29700 1 --2- 172.21.15.90:0/1276792806 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efc5c100d10 0x7efc5c074ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:45.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.923+0000 7efc63b29700 1 --2- 172.21.15.90:0/1276792806 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc5c101f00 0x7efc5c073160 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:45.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.923+0000 7efc63b29700 1 -- 172.21.15.90:0/1276792806 >> 172.21.15.90:0/1276792806 conn(0x7efc5c0fc280 msgr2=0x7efc5c105320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:45.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.923+0000 7efc63b29700 1 -- 172.21.15.90:0/1276792806 shutdown_connections 2024-02-21T23:46:45.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:45.924+0000 7efc63b29700 1 -- 172.21.15.90:0/1276792806 wait complete. 2024-02-21T23:46:45.920 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-21T23:46:46.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.060+0000 7fc69fba4700 1 -- 172.21.15.90:0/3986236393 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6981017d0 msgr2=0x7fc698103bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:46.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.060+0000 7fc69fba4700 1 --2- 172.21.15.90:0/3986236393 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6981017d0 0x7fc698103bc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc694009990 tx=0x7fc694009ca0 comp rx=0 tx=0).stop 2024-02-21T23:46:46.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.060+0000 7fc69fba4700 1 -- 172.21.15.90:0/3986236393 shutdown_connections 2024-02-21T23:46:46.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.060+0000 7fc69fba4700 1 --2- 172.21.15.90:0/3986236393 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc698106a30 0x7fc698108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:46.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.060+0000 7fc69fba4700 1 --2- 172.21.15.90:0/3986236393 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6981017d0 0x7fc698103bc0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:46.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.060+0000 7fc69fba4700 1 --2- 172.21.15.90:0/3986236393 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc698104100 0x7fc6981064f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:46.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.060+0000 7fc69fba4700 1 -- 172.21.15.90:0/3986236393 >> 172.21.15.90:0/3986236393 conn(0x7fc6980fb160 msgr2=0x7fc6980fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:46.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.060+0000 7fc69fba4700 1 -- 172.21.15.90:0/3986236393 shutdown_connections 2024-02-21T23:46:46.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.060+0000 7fc69fba4700 1 -- 172.21.15.90:0/3986236393 wait complete. 2024-02-21T23:46:46.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.061+0000 7fc69fba4700 1 Processor -- start 2024-02-21T23:46:46.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.061+0000 7fc69fba4700 1 -- start start 2024-02-21T23:46:46.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.061+0000 7fc69fba4700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6981017d0 0x7fc698196c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:46.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.061+0000 7fc69fba4700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc698104100 0x7fc6981971b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:46.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.061+0000 7fc69fba4700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc698106a30 0x7fc69819e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:46.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.061+0000 7fc69fba4700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc69819e760 con 0x7fc6981017d0 2024-02-21T23:46:46.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.061+0000 7fc69fba4700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc69819e8d0 con 0x7fc698106a30 2024-02-21T23:46:46.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.061+0000 7fc69fba4700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc69819ea90 con 0x7fc698104100 2024-02-21T23:46:46.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.061+0000 7fc69d940700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6981017d0 0x7fc698196c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:46.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.061+0000 7fc69d940700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6981017d0 0x7fc698196c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49366/0 (socket says 172.21.15.90:49366) 2024-02-21T23:46:46.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.061+0000 7fc69d940700 1 -- 172.21.15.90:0/1694568276 learned_addr learned my addr 172.21.15.90:0/1694568276 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:46.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.061+0000 7fc69d13f700 1 --2- 172.21.15.90:0/1694568276 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc698104100 0x7fc6981971b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:46.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.061+0000 7fc69e141700 1 --2- 172.21.15.90:0/1694568276 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc698106a30 0x7fc69819e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:46.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.062+0000 7fc69d940700 1 -- 172.21.15.90:0/1694568276 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc698104100 msgr2=0x7fc6981971b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:46.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.062+0000 7fc69d940700 1 --2- 172.21.15.90:0/1694568276 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc698104100 0x7fc6981971b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:46.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.062+0000 7fc69d940700 1 -- 172.21.15.90:0/1694568276 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc698106a30 msgr2=0x7fc69819e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:46.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.062+0000 7fc69d940700 1 --2- 172.21.15.90:0/1694568276 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc698106a30 0x7fc69819e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:46.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.062+0000 7fc69d940700 1 -- 172.21.15.90:0/1694568276 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6940095f0 con 0x7fc6981017d0 2024-02-21T23:46:46.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.062+0000 7fc69e141700 1 --2- 172.21.15.90:0/1694568276 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc698106a30 0x7fc69819e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:46:46.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.062+0000 7fc69d940700 1 --2- 172.21.15.90:0/1694568276 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6981017d0 0x7fc698196c70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc694000c00 tx=0x7fc694004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:46.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.062+0000 7fc68effd700 1 -- 172.21.15.90:0/1694568276 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69401d070 con 0x7fc6981017d0 2024-02-21T23:46:46.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.062+0000 7fc68effd700 1 -- 172.21.15.90:0/1694568276 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc6940054f0 con 0x7fc6981017d0 2024-02-21T23:46:46.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.062+0000 7fc69fba4700 1 -- 172.21.15.90:0/1694568276 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc69819ed10 con 0x7fc6981017d0 2024-02-21T23:46:46.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.062+0000 7fc68effd700 1 -- 172.21.15.90:0/1694568276 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc694004cd0 con 0x7fc6981017d0 2024-02-21T23:46:46.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.062+0000 7fc69fba4700 1 -- 172.21.15.90:0/1694568276 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc69819f130 con 0x7fc6981017d0 2024-02-21T23:46:46.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.063+0000 7fc69fba4700 1 -- 172.21.15.90:0/1694568276 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6980fcce0 con 0x7fc6981017d0 2024-02-21T23:46:46.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.064+0000 7fc68effd700 1 -- 172.21.15.90:0/1694568276 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7fc69400fc40 con 0x7fc6981017d0 2024-02-21T23:46:46.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.064+0000 7fc68effd700 1 --2- 172.21.15.90:0/1694568276 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fc6840a11a0 0x7fc6840a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:46.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.064+0000 7fc69d13f700 1 --2- 172.21.15.90:0/1694568276 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fc6840a11a0 0x7fc6840a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:46.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.065+0000 7fc68effd700 1 -- 172.21.15.90:0/1694568276 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fc6940cde10 con 0x7fc6981017d0 2024-02-21T23:46:46.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.065+0000 7fc69d13f700 1 --2- 172.21.15.90:0/1694568276 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fc6840a11a0 0x7fc6840a3660 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc688009c60 tx=0x7fc688009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:46.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.068+0000 7fc68effd700 1 -- 172.21.15.90:0/1694568276 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fc694097440 con 0x7fc6981017d0 2024-02-21T23:46:46.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.267+0000 7fc69fba4700 1 -- 172.21.15.90:0/1694568276 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc69802d0b0 con 0x7fc6981017d0 2024-02-21T23:46:46.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.267+0000 7fc68effd700 1 -- 172.21.15.90:0/1694568276 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc69409aa60 con 0x7fc6981017d0 2024-02-21T23:46:46.253 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-02-21T23:46:46.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.270+0000 7fc69fba4700 1 -- 172.21.15.90:0/1694568276 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fc6840a11a0 msgr2=0x7fc6840a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:46.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.270+0000 7fc69fba4700 1 --2- 172.21.15.90:0/1694568276 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fc6840a11a0 0x7fc6840a3660 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc688009c60 tx=0x7fc688009260 comp rx=0 tx=0).stop 2024-02-21T23:46:46.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.270+0000 7fc69fba4700 1 -- 172.21.15.90:0/1694568276 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6981017d0 msgr2=0x7fc698196c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:46.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.270+0000 7fc69fba4700 1 --2- 172.21.15.90:0/1694568276 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6981017d0 0x7fc698196c70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc694000c00 tx=0x7fc694004730 comp rx=0 tx=0).stop 2024-02-21T23:46:46.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.270+0000 7fc69fba4700 1 -- 172.21.15.90:0/1694568276 shutdown_connections 2024-02-21T23:46:46.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.270+0000 7fc69fba4700 1 --2- 172.21.15.90:0/1694568276 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fc6840a11a0 0x7fc6840a3660 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:46.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.270+0000 7fc69fba4700 1 --2- 172.21.15.90:0/1694568276 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc698106a30 0x7fc69819e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:46.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.270+0000 7fc69fba4700 1 --2- 172.21.15.90:0/1694568276 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6981017d0 0x7fc698196c70 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:46.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.270+0000 7fc69fba4700 1 --2- 172.21.15.90:0/1694568276 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc698104100 0x7fc6981971b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:46.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.270+0000 7fc69fba4700 1 -- 172.21.15.90:0/1694568276 >> 172.21.15.90:0/1694568276 conn(0x7fc6980fb160 msgr2=0x7fc698102230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:46.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.270+0000 7fc69fba4700 1 -- 172.21.15.90:0/1694568276 shutdown_connections 2024-02-21T23:46:46.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.270+0000 7fc69fba4700 1 -- 172.21.15.90:0/1694568276 wait complete. 2024-02-21T23:46:46.267 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-21T23:46:46.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.406+0000 7f2a1c1b2700 1 -- 172.21.15.90:0/147139321 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a14106a40 msgr2=0x7f2a14108e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:46.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.406+0000 7f2a1c1b2700 1 --2- 172.21.15.90:0/147139321 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a14106a40 0x7f2a14108e50 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c00b0a0 tx=0x7f2a0c00b3b0 comp rx=0 tx=0).stop 2024-02-21T23:46:46.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.406+0000 7f2a1c1b2700 1 -- 172.21.15.90:0/147139321 shutdown_connections 2024-02-21T23:46:46.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.406+0000 7f2a1c1b2700 1 --2- 172.21.15.90:0/147139321 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a14106a40 0x7f2a14108e50 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:46.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.406+0000 7f2a1c1b2700 1 --2- 172.21.15.90:0/147139321 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2a141017e0 0x7f2a14103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:46.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.406+0000 7f2a1c1b2700 1 --2- 172.21.15.90:0/147139321 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a14104110 0x7f2a14106500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:46.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.406+0000 7f2a1c1b2700 1 -- 172.21.15.90:0/147139321 >> 172.21.15.90:0/147139321 conn(0x7f2a140fb190 msgr2=0x7f2a140fd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:46.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.406+0000 7f2a1c1b2700 1 -- 172.21.15.90:0/147139321 shutdown_connections 2024-02-21T23:46:46.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.406+0000 7f2a1c1b2700 1 -- 172.21.15.90:0/147139321 wait complete. 2024-02-21T23:46:46.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.407+0000 7f2a1c1b2700 1 Processor -- start 2024-02-21T23:46:46.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.407+0000 7f2a1c1b2700 1 -- start start 2024-02-21T23:46:46.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.407+0000 7f2a1c1b2700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a141017e0 0x7f2a14101100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:46.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.407+0000 7f2a1c1b2700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2a14104110 0x7f2a140ff790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:46.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.407+0000 7f2a1c1b2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a14106a40 0x7f2a140ffcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:46.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.407+0000 7f2a1c1b2700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a14100470 con 0x7f2a14106a40 2024-02-21T23:46:46.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.407+0000 7f2a1c1b2700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a141005e0 con 0x7f2a141017e0 2024-02-21T23:46:46.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.407+0000 7f2a1c1b2700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a141a0dd0 con 0x7f2a14104110 2024-02-21T23:46:46.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.407+0000 7f2a1a74f700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a14106a40 0x7f2a140ffcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:46.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.407+0000 7f2a1a74f700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a14106a40 0x7f2a140ffcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49388/0 (socket says 172.21.15.90:49388) 2024-02-21T23:46:46.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.407+0000 7f2a1a74f700 1 -- 172.21.15.90:0/635994671 learned_addr learned my addr 172.21.15.90:0/635994671 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:46.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.407+0000 7f2a19f4e700 1 --2- 172.21.15.90:0/635994671 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a141017e0 0x7f2a14101100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:46.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.407+0000 7f2a1974d700 1 --2- 172.21.15.90:0/635994671 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2a14104110 0x7f2a140ff790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:46.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.407+0000 7f2a1a74f700 1 -- 172.21.15.90:0/635994671 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2a14104110 msgr2=0x7f2a140ff790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:46.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.407+0000 7f2a1a74f700 1 --2- 172.21.15.90:0/635994671 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2a14104110 0x7f2a140ff790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:46.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.407+0000 7f2a1a74f700 1 -- 172.21.15.90:0/635994671 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a141017e0 msgr2=0x7f2a14101100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:46.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.407+0000 7f2a1a74f700 1 --2- 172.21.15.90:0/635994671 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a141017e0 0x7f2a14101100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:46.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.407+0000 7f2a1a74f700 1 -- 172.21.15.90:0/635994671 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0c009d00 con 0x7f2a14106a40 2024-02-21T23:46:46.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.407+0000 7f2a1a74f700 1 --2- 172.21.15.90:0/635994671 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a14106a40 0x7f2a140ffcd0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c007a60 tx=0x7f2a0c00bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:46.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.407+0000 7f2a06ffd700 1 -- 172.21.15.90:0/635994671 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0c00e040 con 0x7f2a14106a40 2024-02-21T23:46:46.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.407+0000 7f2a06ffd700 1 -- 172.21.15.90:0/635994671 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2a0c0094c0 con 0x7f2a14106a40 2024-02-21T23:46:46.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.407+0000 7f2a1c1b2700 1 -- 172.21.15.90:0/635994671 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a141a1000 con 0x7f2a14106a40 2024-02-21T23:46:46.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.408+0000 7f2a06ffd700 1 -- 172.21.15.90:0/635994671 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0c0052a0 con 0x7f2a14106a40 2024-02-21T23:46:46.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.408+0000 7f2a1c1b2700 1 -- 172.21.15.90:0/635994671 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a141a1480 con 0x7f2a14106a40 2024-02-21T23:46:46.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.408+0000 7f2a1c1b2700 1 -- 172.21.15.90:0/635994671 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a140fcd10 con 0x7f2a14106a40 2024-02-21T23:46:46.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.409+0000 7f2a06ffd700 1 -- 172.21.15.90:0/635994671 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7f2a0c005400 con 0x7f2a14106a40 2024-02-21T23:46:46.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.410+0000 7f2a06ffd700 1 --2- 172.21.15.90:0/635994671 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f2a000a11a0 0x7f2a000a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:46.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.410+0000 7f2a06ffd700 1 -- 172.21.15.90:0/635994671 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f2a0c0ce0a0 con 0x7f2a14106a40 2024-02-21T23:46:46.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.410+0000 7f2a19f4e700 1 --2- 172.21.15.90:0/635994671 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f2a000a11a0 0x7f2a000a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:46.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.410+0000 7f2a19f4e700 1 --2- 172.21.15.90:0/635994671 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f2a000a11a0 0x7f2a000a3660 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2a10005950 tx=0x7f2a1000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:46.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.414+0000 7f2a06ffd700 1 -- 172.21.15.90:0/635994671 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f2a0c0976d0 con 0x7f2a14106a40 2024-02-21T23:46:46.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.615+0000 7f2a1c1b2700 1 -- 172.21.15.90:0/635994671 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2a1402cea0 con 0x7f2a14106a40 2024-02-21T23:46:46.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.616+0000 7f2a06ffd700 1 -- 172.21.15.90:0/635994671 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2a0c09acf0 con 0x7f2a14106a40 2024-02-21T23:46:46.601 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-02-21T23:46:46.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.618+0000 7f2a1c1b2700 1 -- 172.21.15.90:0/635994671 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f2a000a11a0 msgr2=0x7f2a000a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:46.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.618+0000 7f2a1c1b2700 1 --2- 172.21.15.90:0/635994671 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f2a000a11a0 0x7f2a000a3660 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2a10005950 tx=0x7f2a1000b410 comp rx=0 tx=0).stop 2024-02-21T23:46:46.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.618+0000 7f2a1c1b2700 1 -- 172.21.15.90:0/635994671 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a14106a40 msgr2=0x7f2a140ffcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:46.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.618+0000 7f2a1c1b2700 1 --2- 172.21.15.90:0/635994671 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a14106a40 0x7f2a140ffcd0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c007a60 tx=0x7f2a0c00bce0 comp rx=0 tx=0).stop 2024-02-21T23:46:46.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.618+0000 7f2a1c1b2700 1 -- 172.21.15.90:0/635994671 shutdown_connections 2024-02-21T23:46:46.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.618+0000 7f2a1c1b2700 1 --2- 172.21.15.90:0/635994671 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f2a000a11a0 0x7f2a000a3660 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:46.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.618+0000 7f2a1c1b2700 1 --2- 172.21.15.90:0/635994671 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a14106a40 0x7f2a140ffcd0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:46.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.618+0000 7f2a1c1b2700 1 --2- 172.21.15.90:0/635994671 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a141017e0 0x7f2a14101100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:46.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.618+0000 7f2a1c1b2700 1 --2- 172.21.15.90:0/635994671 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2a14104110 0x7f2a140ff790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:46.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.618+0000 7f2a1c1b2700 1 -- 172.21.15.90:0/635994671 >> 172.21.15.90:0/635994671 conn(0x7f2a140fb190 msgr2=0x7f2a14102120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:46.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.618+0000 7f2a1c1b2700 1 -- 172.21.15.90:0/635994671 shutdown_connections 2024-02-21T23:46:46.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.618+0000 7f2a1c1b2700 1 -- 172.21.15.90:0/635994671 wait complete. 2024-02-21T23:46:46.613 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:46 smithi090 ceph-mon[43385]: pgmap v231: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-02-21T23:46:46.614 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:46 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/2540012388' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-21T23:46:46.614 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:46 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/1276792806' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-21T23:46:46.614 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:46 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/1694568276' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-02-21T23:46:46.615 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-02-21T23:46:46.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.755+0000 7fa342c43700 1 -- 172.21.15.90:0/2899633287 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa33c0ffe60 msgr2=0x7fa33c1002e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:46.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.755+0000 7fa342c43700 1 --2- 172.21.15.90:0/2899633287 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa33c0ffe60 0x7fa33c1002e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa328009990 tx=0x7fa328009ca0 comp rx=0 tx=0).stop 2024-02-21T23:46:46.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.755+0000 7fa342c43700 1 -- 172.21.15.90:0/2899633287 shutdown_connections 2024-02-21T23:46:46.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.755+0000 7fa342c43700 1 --2- 172.21.15.90:0/2899633287 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa33c100820 0x7fa33c1080c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:46.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.755+0000 7fa342c43700 1 --2- 172.21.15.90:0/2899633287 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa33c101210 0x7fa33c101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:46.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.755+0000 7fa342c43700 1 --2- 172.21.15.90:0/2899633287 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa33c0ffe60 0x7fa33c1002e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:46.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.755+0000 7fa342c43700 1 -- 172.21.15.90:0/2899633287 >> 172.21.15.90:0/2899633287 conn(0x7fa33c0fa8f0 msgr2=0x7fa33c0fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:46.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.755+0000 7fa342c43700 1 -- 172.21.15.90:0/2899633287 shutdown_connections 2024-02-21T23:46:46.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.755+0000 7fa342c43700 1 -- 172.21.15.90:0/2899633287 wait complete. 2024-02-21T23:46:46.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.756+0000 7fa342c43700 1 Processor -- start 2024-02-21T23:46:46.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.756+0000 7fa342c43700 1 -- start start 2024-02-21T23:46:46.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.756+0000 7fa342c43700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa33c0ffe60 0x7fa33c196cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:46.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.756+0000 7fa342c43700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa33c100820 0x7fa33c197200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:46.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.756+0000 7fa342c43700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa33c101210 0x7fa33c19e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:46.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.756+0000 7fa342c43700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa33c19e7e0 con 0x7fa33c0ffe60 2024-02-21T23:46:46.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.756+0000 7fa342c43700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa33c19e950 con 0x7fa33c101210 2024-02-21T23:46:46.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.756+0000 7fa342c43700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa33c19eb10 con 0x7fa33c100820 2024-02-21T23:46:46.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.756+0000 7fa3409df700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa33c0ffe60 0x7fa33c196cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:46.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.756+0000 7fa3409df700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa33c0ffe60 0x7fa33c196cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49408/0 (socket says 172.21.15.90:49408) 2024-02-21T23:46:46.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.756+0000 7fa3409df700 1 -- 172.21.15.90:0/1582692066 learned_addr learned my addr 172.21.15.90:0/1582692066 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:46.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.756+0000 7fa33bfff700 1 --2- 172.21.15.90:0/1582692066 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa33c100820 0x7fa33c197200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:46.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.756+0000 7fa3411e0700 1 --2- 172.21.15.90:0/1582692066 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa33c101210 0x7fa33c19e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:46.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.756+0000 7fa3409df700 1 -- 172.21.15.90:0/1582692066 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa33c100820 msgr2=0x7fa33c197200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:46.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.756+0000 7fa3409df700 1 --2- 172.21.15.90:0/1582692066 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa33c100820 0x7fa33c197200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:46.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.756+0000 7fa3409df700 1 -- 172.21.15.90:0/1582692066 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa33c101210 msgr2=0x7fa33c19e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:46.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.756+0000 7fa3409df700 1 --2- 172.21.15.90:0/1582692066 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa33c101210 0x7fa33c19e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:46.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.756+0000 7fa3409df700 1 -- 172.21.15.90:0/1582692066 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3280095f0 con 0x7fa33c0ffe60 2024-02-21T23:46:46.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.756+0000 7fa3411e0700 1 --2- 172.21.15.90:0/1582692066 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa33c101210 0x7fa33c19e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:46:46.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.756+0000 7fa33bfff700 1 --2- 172.21.15.90:0/1582692066 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa33c100820 0x7fa33c197200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:46:46.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.757+0000 7fa3409df700 1 --2- 172.21.15.90:0/1582692066 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa33c0ffe60 0x7fa33c196cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa33000d980 tx=0x7fa33000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:46.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.757+0000 7fa339ffb700 1 -- 172.21.15.90:0/1582692066 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33000bb60 con 0x7fa33c0ffe60 2024-02-21T23:46:46.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.757+0000 7fa339ffb700 1 -- 172.21.15.90:0/1582692066 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa330007620 con 0x7fa33c0ffe60 2024-02-21T23:46:46.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.757+0000 7fa342c43700 1 -- 172.21.15.90:0/1582692066 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa33c19edf0 con 0x7fa33c0ffe60 2024-02-21T23:46:46.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.757+0000 7fa339ffb700 1 -- 172.21.15.90:0/1582692066 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33000f620 con 0x7fa33c0ffe60 2024-02-21T23:46:46.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.757+0000 7fa342c43700 1 -- 172.21.15.90:0/1582692066 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa33c19f210 con 0x7fa33c0ffe60 2024-02-21T23:46:46.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.758+0000 7fa342c43700 1 -- 172.21.15.90:0/1582692066 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa33c1070d0 con 0x7fa33c0ffe60 2024-02-21T23:46:46.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.759+0000 7fa339ffb700 1 -- 172.21.15.90:0/1582692066 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7fa330010460 con 0x7fa33c0ffe60 2024-02-21T23:46:46.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.759+0000 7fa339ffb700 1 --2- 172.21.15.90:0/1582692066 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fa32c0a1120 0x7fa32c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:46.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.760+0000 7fa33bfff700 1 --2- 172.21.15.90:0/1582692066 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fa32c0a1120 0x7fa32c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:46.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.760+0000 7fa339ffb700 1 -- 172.21.15.90:0/1582692066 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fa3300cca40 con 0x7fa33c0ffe60 2024-02-21T23:46:46.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.760+0000 7fa33bfff700 1 --2- 172.21.15.90:0/1582692066 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fa32c0a1120 0x7fa32c0a35e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa328006010 tx=0x7fa3280037b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:46.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.764+0000 7fa339ffb700 1 -- 172.21.15.90:0/1582692066 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fa3300cee60 con 0x7fa33c0ffe60 2024-02-21T23:46:46.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:46 smithi176 ceph-mon[47172]: pgmap v231: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-02-21T23:46:46.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:46 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/2540012388' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-21T23:46:46.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:46 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/1276792806' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-21T23:46:46.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:46 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/1694568276' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-02-21T23:46:46.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:46 smithi178 ceph-mon[47547]: pgmap v231: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-02-21T23:46:46.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:46 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/2540012388' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-21T23:46:46.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:46 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/1276792806' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-21T23:46:46.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:46 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/1694568276' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-02-21T23:46:46.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.963+0000 7fa342c43700 1 -- 172.21.15.90:0/1582692066 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fa33c061980 con 0x7fa33c0ffe60 2024-02-21T23:46:46.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.964+0000 7fa339ffb700 1 -- 172.21.15.90:0/1582692066 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v85) v1 ==== 103+0+11 (secure 0 0 0) 0x7fa3300160a0 con 0x7fa33c0ffe60 2024-02-21T23:46:46.949 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-02-21T23:46:46.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.966+0000 7fa342c43700 1 -- 172.21.15.90:0/1582692066 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fa32c0a1120 msgr2=0x7fa32c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:46.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.966+0000 7fa342c43700 1 --2- 172.21.15.90:0/1582692066 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fa32c0a1120 0x7fa32c0a35e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa328006010 tx=0x7fa3280037b0 comp rx=0 tx=0).stop 2024-02-21T23:46:46.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.966+0000 7fa342c43700 1 -- 172.21.15.90:0/1582692066 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa33c0ffe60 msgr2=0x7fa33c196cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:46.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.966+0000 7fa342c43700 1 --2- 172.21.15.90:0/1582692066 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa33c0ffe60 0x7fa33c196cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa33000d980 tx=0x7fa33000dd40 comp rx=0 tx=0).stop 2024-02-21T23:46:46.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.966+0000 7fa342c43700 1 -- 172.21.15.90:0/1582692066 shutdown_connections 2024-02-21T23:46:46.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.966+0000 7fa342c43700 1 --2- 172.21.15.90:0/1582692066 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fa32c0a1120 0x7fa32c0a35e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:46.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.966+0000 7fa342c43700 1 --2- 172.21.15.90:0/1582692066 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa33c101210 0x7fa33c19e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:46.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.966+0000 7fa342c43700 1 --2- 172.21.15.90:0/1582692066 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa33c0ffe60 0x7fa33c196cc0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:46.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.966+0000 7fa342c43700 1 --2- 172.21.15.90:0/1582692066 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa33c100820 0x7fa33c197200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:46.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.966+0000 7fa342c43700 1 -- 172.21.15.90:0/1582692066 >> 172.21.15.90:0/1582692066 conn(0x7fa33c0fa8f0 msgr2=0x7fa33c104210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:46.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.966+0000 7fa342c43700 1 -- 172.21.15.90:0/1582692066 shutdown_connections 2024-02-21T23:46:46.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:46.966+0000 7fa342c43700 1 -- 172.21.15.90:0/1582692066 wait complete. 2024-02-21T23:46:46.963 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-02-21T23:46:47.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.101+0000 7f437ee83700 1 -- 172.21.15.90:0/3416146615 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43780ffc10 msgr2=0x7f4378100030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:47.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.101+0000 7f437ee83700 1 --2- 172.21.15.90:0/3416146615 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43780ffc10 0x7f4378100030 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f436c0099c0 tx=0x7f436c009cd0 comp rx=0 tx=0).stop 2024-02-21T23:46:47.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.101+0000 7f437ee83700 1 -- 172.21.15.90:0/3416146615 shutdown_connections 2024-02-21T23:46:47.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.101+0000 7f437ee83700 1 --2- 172.21.15.90:0/3416146615 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43781017a0 0x7f4378108050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:47.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.101+0000 7f437ee83700 1 --2- 172.21.15.90:0/3416146615 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43780ffc10 0x7f4378100030 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:47.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.101+0000 7f437ee83700 1 --2- 172.21.15.90:0/3416146615 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4378100e00 0x7f4378101260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:47.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.101+0000 7f437ee83700 1 -- 172.21.15.90:0/3416146615 >> 172.21.15.90:0/3416146615 conn(0x7f43780fb180 msgr2=0x7f43780fd5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:47.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.101+0000 7f437ee83700 1 -- 172.21.15.90:0/3416146615 shutdown_connections 2024-02-21T23:46:47.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.101+0000 7f437ee83700 1 -- 172.21.15.90:0/3416146615 wait complete. 2024-02-21T23:46:47.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.101+0000 7f437ee83700 1 Processor -- start 2024-02-21T23:46:47.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.101+0000 7f437ee83700 1 -- start start 2024-02-21T23:46:47.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.102+0000 7f437ee83700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f43780ffc10 0x7f4378198ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:47.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.102+0000 7f437ee83700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4378100e00 0x7f4378199400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:47.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.102+0000 7f437ee83700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43781017a0 0x7f43781a03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:47.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.102+0000 7f437ee83700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43781a09a0 con 0x7f43781017a0 2024-02-21T23:46:47.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.102+0000 7f437ee83700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43781a0ae0 con 0x7f4378100e00 2024-02-21T23:46:47.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.102+0000 7f437ee83700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43781a0c70 con 0x7f43780ffc10 2024-02-21T23:46:47.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.102+0000 7f437d420700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43781017a0 0x7f43781a03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:47.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.102+0000 7f437d420700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43781017a0 0x7f43781a03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49432/0 (socket says 172.21.15.90:49432) 2024-02-21T23:46:47.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.102+0000 7f437d420700 1 -- 172.21.15.90:0/2917286309 learned_addr learned my addr 172.21.15.90:0/2917286309 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:47.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.102+0000 7f4377fff700 1 --2- 172.21.15.90:0/2917286309 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4378100e00 0x7f4378199400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:47.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.102+0000 7f437d420700 1 -- 172.21.15.90:0/2917286309 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f43780ffc10 msgr2=0x7f4378198ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:47.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.102+0000 7f437d420700 1 --2- 172.21.15.90:0/2917286309 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f43780ffc10 0x7f4378198ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:47.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.102+0000 7f437d420700 1 -- 172.21.15.90:0/2917286309 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4378100e00 msgr2=0x7f4378199400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:47.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.102+0000 7f437d420700 1 --2- 172.21.15.90:0/2917286309 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4378100e00 0x7f4378199400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:47.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.102+0000 7f437d420700 1 -- 172.21.15.90:0/2917286309 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f436c0095f0 con 0x7f43781017a0 2024-02-21T23:46:47.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.102+0000 7f437d420700 1 --2- 172.21.15.90:0/2917286309 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43781017a0 0x7f43781a03d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4370007ee0 tx=0x7f437000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:47.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.102+0000 7f4375ffb700 1 -- 172.21.15.90:0/2917286309 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4370017070 con 0x7f43781017a0 2024-02-21T23:46:47.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.102+0000 7f4375ffb700 1 -- 172.21.15.90:0/2917286309 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f437000f040 con 0x7f43781017a0 2024-02-21T23:46:47.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.102+0000 7f437ee83700 1 -- 172.21.15.90:0/2917286309 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43781a0f50 con 0x7f43781017a0 2024-02-21T23:46:47.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.102+0000 7f4375ffb700 1 -- 172.21.15.90:0/2917286309 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4370005740 con 0x7f43781017a0 2024-02-21T23:46:47.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.103+0000 7f437ee83700 1 -- 172.21.15.90:0/2917286309 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43781a1380 con 0x7f43781017a0 2024-02-21T23:46:47.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.103+0000 7f437ee83700 1 -- 172.21.15.90:0/2917286309 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43780671f0 con 0x7f43781017a0 2024-02-21T23:46:47.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.104+0000 7f4375ffb700 1 -- 172.21.15.90:0/2917286309 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7f43700058a0 con 0x7f43781017a0 2024-02-21T23:46:47.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.105+0000 7f4375ffb700 1 --2- 172.21.15.90:0/2917286309 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f43680a1120 0x7f43680a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:47.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.105+0000 7f437cc1f700 1 --2- 172.21.15.90:0/2917286309 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f43680a1120 0x7f43680a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:47.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.105+0000 7f4375ffb700 1 -- 172.21.15.90:0/2917286309 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f43700cc390 con 0x7f43781017a0 2024-02-21T23:46:47.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.105+0000 7f437cc1f700 1 --2- 172.21.15.90:0/2917286309 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f43680a1120 0x7f43680a35e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f436c0058e0 tx=0x7f436c0036f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:47.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.109+0000 7f4375ffb700 1 -- 172.21.15.90:0/2917286309 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f4370095930 con 0x7f43781017a0 2024-02-21T23:46:47.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.328+0000 7f437ee83700 1 -- 172.21.15.90:0/2917286309 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f43781a1660 con 0x7f43781017a0 2024-02-21T23:46:47.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.328+0000 7f4375ffb700 1 -- 172.21.15.90:0/2917286309 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v85) v1 ==== 81+0+591 (secure 0 0 0) 0x7f4370098f50 con 0x7f43781017a0 2024-02-21T23:46:47.313 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/159655312 2024-02-22T23:41:33.360021+0000 2024-02-21T23:46:47.313 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6801/1385551604 2024-02-22T23:41:33.360021+0000 2024-02-21T23:46:47.314 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/1452574770 2024-02-22T23:41:33.360021+0000 2024-02-21T23:46:47.314 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/3815292343 2024-02-22T23:41:11.484145+0000 2024-02-21T23:46:47.314 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6800/1385551604 2024-02-22T23:41:33.360021+0000 2024-02-21T23:46:47.314 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6801/4247695574 2024-02-22T23:41:11.484145+0000 2024-02-21T23:46:47.314 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/1328266622 2024-02-22T23:41:11.484145+0000 2024-02-21T23:46:47.314 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6800/4247695574 2024-02-22T23:41:11.484145+0000 2024-02-21T23:46:47.314 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/2569421522 2024-02-22T23:41:11.484145+0000 2024-02-21T23:46:47.314 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/3019701027 2024-02-22T23:41:33.360021+0000 2024-02-21T23:46:47.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.330+0000 7f437ee83700 1 -- 172.21.15.90:0/2917286309 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f43680a1120 msgr2=0x7f43680a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:47.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.330+0000 7f437ee83700 1 --2- 172.21.15.90:0/2917286309 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f43680a1120 0x7f43680a35e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f436c0058e0 tx=0x7f436c0036f0 comp rx=0 tx=0).stop 2024-02-21T23:46:47.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.330+0000 7f437ee83700 1 -- 172.21.15.90:0/2917286309 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43781017a0 msgr2=0x7f43781a03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:47.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.330+0000 7f437ee83700 1 --2- 172.21.15.90:0/2917286309 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43781017a0 0x7f43781a03d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4370007ee0 tx=0x7f437000d3b0 comp rx=0 tx=0).stop 2024-02-21T23:46:47.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.330+0000 7f437ee83700 1 -- 172.21.15.90:0/2917286309 shutdown_connections 2024-02-21T23:46:47.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.330+0000 7f437ee83700 1 --2- 172.21.15.90:0/2917286309 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f43680a1120 0x7f43680a35e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:47.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.330+0000 7f437ee83700 1 --2- 172.21.15.90:0/2917286309 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43781017a0 0x7f43781a03d0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:47.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.330+0000 7f437ee83700 1 --2- 172.21.15.90:0/2917286309 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f43780ffc10 0x7f4378198ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:47.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.330+0000 7f437ee83700 1 --2- 172.21.15.90:0/2917286309 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4378100e00 0x7f4378199400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:47.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.330+0000 7f437ee83700 1 -- 172.21.15.90:0/2917286309 >> 172.21.15.90:0/2917286309 conn(0x7f43780fb180 msgr2=0x7f4378068f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:47.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.331+0000 7f437ee83700 1 -- 172.21.15.90:0/2917286309 shutdown_connections 2024-02-21T23:46:47.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.331+0000 7f437ee83700 1 -- 172.21.15.90:0/2917286309 wait complete. 2024-02-21T23:46:47.317 INFO:teuthology.orchestra.run.smithi090.stderr:listed 10 entries 2024-02-21T23:46:47.328 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-21T23:46:47.328 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:46:47.346 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:46:47.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.525+0000 7fde7ab74700 1 -- 172.21.15.90:0/2631678454 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde74100c10 msgr2=0x7fde74101030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:47.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.525+0000 7fde7ab74700 1 --2- 172.21.15.90:0/2631678454 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde74100c10 0x7fde74101030 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fde680099c0 tx=0x7fde68009cd0 comp rx=0 tx=0).stop 2024-02-21T23:46:47.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.525+0000 7fde7ab74700 1 -- 172.21.15.90:0/2631678454 shutdown_connections 2024-02-21T23:46:47.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.525+0000 7fde7ab74700 1 --2- 172.21.15.90:0/2631678454 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde741027a0 0x7fde74109050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:47.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.525+0000 7fde7ab74700 1 --2- 172.21.15.90:0/2631678454 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde74100c10 0x7fde74101030 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:47.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.525+0000 7fde7ab74700 1 --2- 172.21.15.90:0/2631678454 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde74101e00 0x7fde74102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:47.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.525+0000 7fde7ab74700 1 -- 172.21.15.90:0/2631678454 >> 172.21.15.90:0/2631678454 conn(0x7fde740fc180 msgr2=0x7fde740fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:47.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.525+0000 7fde7ab74700 1 -- 172.21.15.90:0/2631678454 shutdown_connections 2024-02-21T23:46:47.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.525+0000 7fde7ab74700 1 -- 172.21.15.90:0/2631678454 wait complete. 2024-02-21T23:46:47.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.526+0000 7fde7ab74700 1 Processor -- start 2024-02-21T23:46:47.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.526+0000 7fde7ab74700 1 -- start start 2024-02-21T23:46:47.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.526+0000 7fde7ab74700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde74100c10 0x7fde74196ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:47.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.526+0000 7fde7ab74700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde74101e00 0x7fde74196ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:47.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.526+0000 7fde7ab74700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde741027a0 0x7fde7419e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:47.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.526+0000 7fde7ab74700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7419e5e0 con 0x7fde741027a0 2024-02-21T23:46:47.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.526+0000 7fde7ab74700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7419e720 con 0x7fde74100c10 2024-02-21T23:46:47.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.526+0000 7fde7ab74700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7419e8b0 con 0x7fde74101e00 2024-02-21T23:46:47.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.527+0000 7fde79111700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde741027a0 0x7fde7419e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:47.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.527+0000 7fde79111700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde741027a0 0x7fde7419e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49440/0 (socket says 172.21.15.90:49440) 2024-02-21T23:46:47.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.527+0000 7fde79111700 1 -- 172.21.15.90:0/3515830779 learned_addr learned my addr 172.21.15.90:0/3515830779 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:47.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.527+0000 7fde73fff700 1 --2- 172.21.15.90:0/3515830779 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde74101e00 0x7fde74196ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:47.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.527+0000 7fde78910700 1 --2- 172.21.15.90:0/3515830779 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde74100c10 0x7fde74196ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:47.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.527+0000 7fde79111700 1 -- 172.21.15.90:0/3515830779 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde74101e00 msgr2=0x7fde74196ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:47.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.527+0000 7fde79111700 1 --2- 172.21.15.90:0/3515830779 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde74101e00 0x7fde74196ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:47.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.527+0000 7fde79111700 1 -- 172.21.15.90:0/3515830779 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde74100c10 msgr2=0x7fde74196ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:47.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.527+0000 7fde79111700 1 --2- 172.21.15.90:0/3515830779 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde74100c10 0x7fde74196ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:47.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.527+0000 7fde79111700 1 -- 172.21.15.90:0/3515830779 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde680095f0 con 0x7fde741027a0 2024-02-21T23:46:47.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.527+0000 7fde78910700 1 --2- 172.21.15.90:0/3515830779 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde74100c10 0x7fde74196ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:46:47.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.527+0000 7fde79111700 1 --2- 172.21.15.90:0/3515830779 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde741027a0 0x7fde7419e010 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fde6c007d70 tx=0x7fde6c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:47.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.527+0000 7fde71ffb700 1 -- 172.21.15.90:0/3515830779 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6c00ad40 con 0x7fde741027a0 2024-02-21T23:46:47.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.527+0000 7fde71ffb700 1 -- 172.21.15.90:0/3515830779 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fde6c0128e0 con 0x7fde741027a0 2024-02-21T23:46:47.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.527+0000 7fde7ab74700 1 -- 172.21.15.90:0/3515830779 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde7419eb90 con 0x7fde741027a0 2024-02-21T23:46:47.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.527+0000 7fde71ffb700 1 -- 172.21.15.90:0/3515830779 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6c0116b0 con 0x7fde741027a0 2024-02-21T23:46:47.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.527+0000 7fde7ab74700 1 -- 172.21.15.90:0/3515830779 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde7419f0a0 con 0x7fde741027a0 2024-02-21T23:46:47.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.528+0000 7fde7ab74700 1 -- 172.21.15.90:0/3515830779 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde74107f60 con 0x7fde741027a0 2024-02-21T23:46:47.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.529+0000 7fde71ffb700 1 -- 172.21.15.90:0/3515830779 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7fde6c012420 con 0x7fde741027a0 2024-02-21T23:46:47.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.530+0000 7fde71ffb700 1 --2- 172.21.15.90:0/3515830779 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fde640a11a0 0x7fde640a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:47.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.530+0000 7fde78910700 1 --2- 172.21.15.90:0/3515830779 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fde640a11a0 0x7fde640a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:47.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.530+0000 7fde71ffb700 1 -- 172.21.15.90:0/3515830779 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fde6c0ccd10 con 0x7fde741027a0 2024-02-21T23:46:47.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.530+0000 7fde78910700 1 --2- 172.21.15.90:0/3515830779 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fde640a11a0 0x7fde640a3660 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fde6800b580 tx=0x7fde68005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:47.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.534+0000 7fde71ffb700 1 -- 172.21.15.90:0/3515830779 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fde6c0962b0 con 0x7fde741027a0 2024-02-21T23:46:47.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.739+0000 7fde7ab74700 1 -- 172.21.15.90:0/3515830779 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde740679a0 con 0x7fde741027a0 2024-02-21T23:46:47.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.741+0000 7fde71ffb700 1 -- 172.21.15.90:0/3515830779 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23925 (secure 0 0 0) 0x7fde6c0998d0 con 0x7fde741027a0 2024-02-21T23:46:47.726 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:46:47.726 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":85,"fsid":"7ed81218-d112-11ee-95bd-87774f69a715","created":"2024-02-21T23:40:41.785526+0000","modified":"2024-02-21T23:46:38.878698+0000","last_up_change":"2024-02-21T23:46:00.860501+0000","last_in_change":"2024-02-21T23:45:44.423815+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:43:21.675789+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:46:32.337047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-21T23:46:33.622440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-21T23:46:36.686123+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"77878890-d9a5-4d5b-a943-fec29ee0446e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6803","nonce":885319826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6805","nonce":885319826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6809","nonce":885319826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6807","nonce":885319826}]},"public_addr":"172.21.15.90:6803/885319826","cluster_addr":"172.21.15.90:6805/885319826","heartbeat_back_addr":"172.21.15.90:6809/885319826","heartbeat_front_addr":"172.21.15.90:6807/885319826","state":["exists","up"]},{"osd":1,"uuid":"7d6ea950-7a57-4f13-ab38-10a658912783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6801","nonce":4085240925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6803","nonce":4085240925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6807","nonce":4085240925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6805","nonce":4085240925}]},"public_addr":"172.21.15.176:6801/4085240925","cluster_addr":"172.21.15.176:6803/4085240925","heartbeat_back_addr":"172.21.15.176:6807/4085240925","heartbeat_front_addr":"172.21.15.176:6805/4085240925","state":["exists","up"]},{"osd":2,"uuid":"b83486bb-27f2-4d56-97fb-fb336fec4b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6801","nonce":3597250789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6803","nonce":3597250789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6807","nonce":3597250789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6805","nonce":3597250789}]},"public_addr":"172.21.15.178:6801/3597250789","cluster_addr":"172.21.15.178:6803/3597250789","heartbeat_back_addr":"172.21.15.178:6807/3597250789","heartbeat_front_addr":"172.21.15.178:6805/3597250789","state":["exists","up"]},{"osd":3,"uuid":"870e5bb6-eed5-4a0a-8e03-38de4f816b44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6811","nonce":3299043821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6813","nonce":3299043821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6817","nonce":3299043821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6815","nonce":3299043821}]},"public_addr":"172.21.15.90:6811/3299043821","cluster_addr":"172.21.15.90:6813/3299043821","heartbeat_back_addr":"172.21.15.90:6817/3299043821","heartbeat_front_addr":"172.21.15.90:6815/3299043821","state":["exists","up"]},{"osd":4,"uuid":"cb9f7f48-76e0-4368-ab87-2a09a8487ccd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6809","nonce":3548915687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6811","nonce":3548915687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6815","nonce":3548915687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6813","nonce":3548915687}]},"public_addr":"172.21.15.176:6809/3548915687","cluster_addr":"172.21.15.176:6811/3548915687","heartbeat_back_addr":"172.21.15.176:6815/3548915687","heartbeat_front_addr":"172.21.15.176:6813/3548915687","state":["exists","up"]},{"osd":5,"uuid":"54749022-c50d-4831-a1d6-8412b59c8a01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6809","nonce":286274518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6811","nonce":286274518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6815","nonce":286274518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6813","nonce":286274518}]},"public_addr":"172.21.15.178:6809/286274518","cluster_addr":"172.21.15.178:6811/286274518","heartbeat_back_addr":"172.21.15.178:6815/286274518","heartbeat_front_addr":"172.21.15.178:6813/286274518","state":["exists","up"]},{"osd":6,"uuid":"09bb5393-2efd-4a63-b0e8-99bef5bced13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6819","nonce":67640620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6821","nonce":67640620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6825","nonce":67640620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6823","nonce":67640620}]},"public_addr":"172.21.15.90:6819/67640620","cluster_addr":"172.21.15.90:6821/67640620","heartbeat_back_addr":"172.21.15.90:6825/67640620","heartbeat_front_addr":"172.21.15.90:6823/67640620","state":["exists","up"]},{"osd":7,"uuid":"7a859e69-6b63-467b-a3b3-99247d66f5c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6817","nonce":3998561598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6819","nonce":3998561598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6823","nonce":3998561598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6821","nonce":3998561598}]},"public_addr":"172.21.15.176:6817/3998561598","cluster_addr":"172.21.15.176:6819/3998561598","heartbeat_back_addr":"172.21.15.176:6823/3998561598","heartbeat_front_addr":"172.21.15.176:6821/3998561598","state":["exists","up"]},{"osd":8,"uuid":"bc23144e-22a5-4894-a8b5-9323383574d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6817","nonce":490229963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6819","nonce":490229963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6823","nonce":490229963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6821","nonce":490229963}]},"public_addr":"172.21.15.178:6817/490229963","cluster_addr":"172.21.15.178:6819/490229963","heartbeat_back_addr":"172.21.15.178:6823/490229963","heartbeat_front_addr":"172.21.15.178:6821/490229963","state":["exists","up"]},{"osd":9,"uuid":"0c2cecd4-55de-4fc4-a6f4-ca97c8460444","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6827","nonce":2061394761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6829","nonce":2061394761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6833","nonce":2061394761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6831","nonce":2061394761}]},"public_addr":"172.21.15.90:6827/2061394761","cluster_addr":"172.21.15.90:6829/2061394761","heartbeat_back_addr":"172.21.15.90:6833/2061394761","heartbeat_front_addr":"172.21.15.90:6831/2061394761","state":["exists","up"]},{"osd":10,"uuid":"fa04232b-b1ec-447a-8e94-7c9bf26d0291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6825","nonce":639411757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6827","nonce":639411757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6831","nonce":639411757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6829","nonce":639411757}]},"public_addr":"172.21.15.176:6825/639411757","cluster_addr":"172.21.15.176:6827/639411757","heartbeat_back_addr":"172.21.15.176:6831/639411757","heartbeat_front_addr":"172.21.15.176:6829/639411757","state":["exists","up"]},{"osd":11,"uuid":"7282e294-64fc-4856-9c74-86115f69408c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6825","nonce":4144536256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6827","nonce":4144536256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6831","nonce":4144536256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6829","nonce":4144536256}]},"public_addr":"172.21.15.178:6825/4144536256","cluster_addr":"172.21.15.178:6827/4144536256","heartbeat_back_addr":"172.21.15.178:6831/4144536256","heartbeat_front_addr":"172.21.15.178:6829/4144536256","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:42:44.280345+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:01.235630+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:19.408896+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:36.201749+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:53.401854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:44:11.342158+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:44:29.324502+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:44:46.821595+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:04.642344+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:23.019004+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:41.140149+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:59.401951+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/159655312":"2024-02-22T23:41:33.360021+0000","172.21.15.90:6801/1385551604":"2024-02-22T23:41:33.360021+0000","172.21.15.90:0/1452574770":"2024-02-22T23:41:33.360021+0000","172.21.15.90:0/3815292343":"2024-02-22T23:41:11.484145+0000","172.21.15.90:6800/1385551604":"2024-02-22T23:41:33.360021+0000","172.21.15.90:6801/4247695574":"2024-02-22T23:41:11.484145+0000","172.21.15.90:0/1328266622":"2024-02-22T23:41:11.484145+0000","172.21.15.90:6800/4247695574":"2024-02-22T23:41:11.484145+0000","172.21.15.90:0/2569421522":"2024-02-22T23:41:11.484145+0000","172.21.15.90:0/3019701027":"2024-02-22T23:41:33.360021+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:46:47.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:47 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/635994671' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-02-21T23:46:47.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:47 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/1582692066' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-02-21T23:46:47.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:47 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/2917286309' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-21T23:46:47.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.743+0000 7fde7ab74700 1 -- 172.21.15.90:0/3515830779 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fde640a11a0 msgr2=0x7fde640a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:47.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.743+0000 7fde7ab74700 1 --2- 172.21.15.90:0/3515830779 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fde640a11a0 0x7fde640a3660 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fde6800b580 tx=0x7fde68005fb0 comp rx=0 tx=0).stop 2024-02-21T23:46:47.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.743+0000 7fde7ab74700 1 -- 172.21.15.90:0/3515830779 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde741027a0 msgr2=0x7fde7419e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:47.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.743+0000 7fde7ab74700 1 --2- 172.21.15.90:0/3515830779 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde741027a0 0x7fde7419e010 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fde6c007d70 tx=0x7fde6c00a340 comp rx=0 tx=0).stop 2024-02-21T23:46:47.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.743+0000 7fde7ab74700 1 -- 172.21.15.90:0/3515830779 shutdown_connections 2024-02-21T23:46:47.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.743+0000 7fde7ab74700 1 --2- 172.21.15.90:0/3515830779 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fde640a11a0 0x7fde640a3660 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:47.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.743+0000 7fde7ab74700 1 --2- 172.21.15.90:0/3515830779 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde741027a0 0x7fde7419e010 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:47.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.743+0000 7fde7ab74700 1 --2- 172.21.15.90:0/3515830779 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde74100c10 0x7fde74196ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:47.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.743+0000 7fde7ab74700 1 --2- 172.21.15.90:0/3515830779 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde74101e00 0x7fde74196ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:47.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.743+0000 7fde7ab74700 1 -- 172.21.15.90:0/3515830779 >> 172.21.15.90:0/3515830779 conn(0x7fde740fc180 msgr2=0x7fde741050a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:47.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.743+0000 7fde7ab74700 1 -- 172.21.15.90:0/3515830779 shutdown_connections 2024-02-21T23:46:47.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.743+0000 7fde7ab74700 1 -- 172.21.15.90:0/3515830779 wait complete. 2024-02-21T23:46:47.740 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:43:21.675789+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-21T23:46:32.337047+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-21T23:46:33.622440+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-02-21T23:46:36.686123+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-02-21T23:46:47.741 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:46:47.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:47 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/635994671' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-02-21T23:46:47.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:47 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/1582692066' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-02-21T23:46:47.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:47 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/2917286309' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-21T23:46:47.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.883+0000 7f35234f5700 1 -- 172.21.15.90:0/1778393408 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f351c101f00 msgr2=0x7f351c102360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.883+0000 7f35234f5700 1 --2- 172.21.15.90:0/1778393408 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f351c101f00 0x7f351c102360 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f35180099c0 tx=0x7f3518009cd0 comp rx=0 tx=0).stop 2024-02-21T23:46:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.883+0000 7f35234f5700 1 -- 172.21.15.90:0/1778393408 shutdown_connections 2024-02-21T23:46:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.883+0000 7f35234f5700 1 --2- 172.21.15.90:0/1778393408 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f351c1028a0 0x7f351c109150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.883+0000 7f35234f5700 1 --2- 172.21.15.90:0/1778393408 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f351c100d10 0x7f351c101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.883+0000 7f35234f5700 1 --2- 172.21.15.90:0/1778393408 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f351c101f00 0x7f351c102360 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.883+0000 7f35234f5700 1 -- 172.21.15.90:0/1778393408 >> 172.21.15.90:0/1778393408 conn(0x7f351c0fc280 msgr2=0x7f351c0fe6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.883+0000 7f35234f5700 1 -- 172.21.15.90:0/1778393408 shutdown_connections 2024-02-21T23:46:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.883+0000 7f35234f5700 1 -- 172.21.15.90:0/1778393408 wait complete. 2024-02-21T23:46:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.883+0000 7f35234f5700 1 Processor -- start 2024-02-21T23:46:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.883+0000 7f35234f5700 1 -- start start 2024-02-21T23:46:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.884+0000 7f35234f5700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f351c100d10 0x7f351c196cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.884+0000 7f35234f5700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f351c101f00 0x7f351c197200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.884+0000 7f35234f5700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f351c1028a0 0x7f351c19e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.884+0000 7f35234f5700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351c19e7e0 con 0x7f351c100d10 2024-02-21T23:46:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.884+0000 7f35234f5700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351c19e950 con 0x7f351c101f00 2024-02-21T23:46:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.884+0000 7f35234f5700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351c19eb10 con 0x7f351c1028a0 2024-02-21T23:46:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.884+0000 7f3521291700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f351c100d10 0x7f351c196cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.884+0000 7f3521291700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f351c100d10 0x7f351c196cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49462/0 (socket says 172.21.15.90:49462) 2024-02-21T23:46:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.884+0000 7f3521291700 1 -- 172.21.15.90:0/4180373811 learned_addr learned my addr 172.21.15.90:0/4180373811 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.884+0000 7f3521a92700 1 --2- 172.21.15.90:0/4180373811 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f351c1028a0 0x7f351c19e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.884+0000 7f3520a90700 1 --2- 172.21.15.90:0/4180373811 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f351c101f00 0x7f351c197200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.884+0000 7f3521291700 1 -- 172.21.15.90:0/4180373811 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f351c1028a0 msgr2=0x7f351c19e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.884+0000 7f3521291700 1 --2- 172.21.15.90:0/4180373811 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f351c1028a0 0x7f351c19e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.884+0000 7f3521291700 1 -- 172.21.15.90:0/4180373811 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f351c101f00 msgr2=0x7f351c197200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.884+0000 7f3521291700 1 --2- 172.21.15.90:0/4180373811 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f351c101f00 0x7f351c197200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.884+0000 7f3521291700 1 -- 172.21.15.90:0/4180373811 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35180095f0 con 0x7f351c100d10 2024-02-21T23:46:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.884+0000 7f3521291700 1 --2- 172.21.15.90:0/4180373811 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f351c100d10 0x7f351c196cc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f350c00c970 tx=0x7f350c00cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.885+0000 7f35127fc700 1 -- 172.21.15.90:0/4180373811 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350c004220 con 0x7f351c100d10 2024-02-21T23:46:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.885+0000 7f35127fc700 1 -- 172.21.15.90:0/4180373811 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f350c007d60 con 0x7f351c100d10 2024-02-21T23:46:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.885+0000 7f35234f5700 1 -- 172.21.15.90:0/4180373811 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f351c19edf0 con 0x7f351c100d10 2024-02-21T23:46:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.885+0000 7f35127fc700 1 -- 172.21.15.90:0/4180373811 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350c003d00 con 0x7f351c100d10 2024-02-21T23:46:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.885+0000 7f35234f5700 1 -- 172.21.15.90:0/4180373811 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f351c19f270 con 0x7f351c100d10 2024-02-21T23:46:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.885+0000 7f35234f5700 1 -- 172.21.15.90:0/4180373811 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f351c108190 con 0x7f351c100d10 2024-02-21T23:46:47.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.887+0000 7f35127fc700 1 -- 172.21.15.90:0/4180373811 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7f350c004380 con 0x7f351c100d10 2024-02-21T23:46:47.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.887+0000 7f35127fc700 1 --2- 172.21.15.90:0/4180373811 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f35080a11a0 0x7f35080a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:47.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.887+0000 7f3520a90700 1 --2- 172.21.15.90:0/4180373811 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f35080a11a0 0x7f35080a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:47.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.887+0000 7f35127fc700 1 -- 172.21.15.90:0/4180373811 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f350c0cca10 con 0x7f351c100d10 2024-02-21T23:46:47.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.888+0000 7f3520a90700 1 --2- 172.21.15.90:0/4180373811 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f35080a11a0 0x7f35080a3660 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3518000c00 tx=0x7f3518005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:47.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:47.891+0000 7f35127fc700 1 -- 172.21.15.90:0/4180373811 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f350c096030 con 0x7f351c100d10 2024-02-21T23:46:47.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:47 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/635994671' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-02-21T23:46:47.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:47 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/1582692066' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-02-21T23:46:47.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:47 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/2917286309' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-21T23:46:48.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.084+0000 7f35234f5700 1 -- 172.21.15.90:0/4180373811 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f351c02cc70 con 0x7f351c100d10 2024-02-21T23:46:48.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.085+0000 7f35127fc700 1 -- 172.21.15.90:0/4180373811 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7f350c099650 con 0x7f351c100d10 2024-02-21T23:46:48.070 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-02-21T23:46:48.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.087+0000 7f35234f5700 1 -- 172.21.15.90:0/4180373811 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f35080a11a0 msgr2=0x7f35080a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:48.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.087+0000 7f35234f5700 1 --2- 172.21.15.90:0/4180373811 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f35080a11a0 0x7f35080a3660 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3518000c00 tx=0x7f3518005fb0 comp rx=0 tx=0).stop 2024-02-21T23:46:48.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.087+0000 7f35234f5700 1 -- 172.21.15.90:0/4180373811 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f351c100d10 msgr2=0x7f351c196cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:48.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.087+0000 7f35234f5700 1 --2- 172.21.15.90:0/4180373811 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f351c100d10 0x7f351c196cc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f350c00c970 tx=0x7f350c00cd30 comp rx=0 tx=0).stop 2024-02-21T23:46:48.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.087+0000 7f35234f5700 1 -- 172.21.15.90:0/4180373811 shutdown_connections 2024-02-21T23:46:48.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.087+0000 7f35234f5700 1 --2- 172.21.15.90:0/4180373811 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f35080a11a0 0x7f35080a3660 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:48.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.087+0000 7f35234f5700 1 --2- 172.21.15.90:0/4180373811 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f351c1028a0 0x7f351c19e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:48.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.087+0000 7f35234f5700 1 --2- 172.21.15.90:0/4180373811 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f351c100d10 0x7f351c196cc0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:48.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.087+0000 7f35234f5700 1 --2- 172.21.15.90:0/4180373811 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f351c101f00 0x7f351c197200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:48.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.087+0000 7f35234f5700 1 -- 172.21.15.90:0/4180373811 >> 172.21.15.90:0/4180373811 conn(0x7f351c0fc280 msgr2=0x7f351c1052d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:48.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.087+0000 7f35234f5700 1 -- 172.21.15.90:0/4180373811 shutdown_connections 2024-02-21T23:46:48.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.087+0000 7f35234f5700 1 -- 172.21.15.90:0/4180373811 wait complete. 2024-02-21T23:46:48.084 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-21T23:46:48.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.223+0000 7fdf3d9c0700 1 -- 172.21.15.90:0/1666390916 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf38100860 msgr2=0x7fdf38108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:48.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.223+0000 7fdf3d9c0700 1 --2- 172.21.15.90:0/1666390916 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf38100860 0x7fdf38108120 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fdf3000b0a0 tx=0x7fdf3000b3b0 comp rx=0 tx=0).stop 2024-02-21T23:46:48.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.223+0000 7fdf3d9c0700 1 -- 172.21.15.90:0/1666390916 shutdown_connections 2024-02-21T23:46:48.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.223+0000 7fdf3d9c0700 1 --2- 172.21.15.90:0/1666390916 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf38100860 0x7fdf38108120 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:48.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.223+0000 7fdf3d9c0700 1 --2- 172.21.15.90:0/1666390916 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdf38101190 0x7fdf381015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:48.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.223+0000 7fdf3d9c0700 1 --2- 172.21.15.90:0/1666390916 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf380ffec0 0x7fdf38100320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:48.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.223+0000 7fdf3d9c0700 1 -- 172.21.15.90:0/1666390916 >> 172.21.15.90:0/1666390916 conn(0x7fdf380fa8f0 msgr2=0x7fdf380fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:48.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.223+0000 7fdf3d9c0700 1 -- 172.21.15.90:0/1666390916 shutdown_connections 2024-02-21T23:46:48.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.223+0000 7fdf3d9c0700 1 -- 172.21.15.90:0/1666390916 wait complete. 2024-02-21T23:46:48.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.223+0000 7fdf3d9c0700 1 Processor -- start 2024-02-21T23:46:48.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.223+0000 7fdf3d9c0700 1 -- start start 2024-02-21T23:46:48.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.224+0000 7fdf3d9c0700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf380ffec0 0x7fdf38196de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:48.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.224+0000 7fdf3d9c0700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf38100860 0x7fdf38197320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:48.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.224+0000 7fdf3d9c0700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdf38101190 0x7fdf3819e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:48.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.224+0000 7fdf3d9c0700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3819e880 con 0x7fdf380ffec0 2024-02-21T23:46:48.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.224+0000 7fdf3d9c0700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3819e9f0 con 0x7fdf38100860 2024-02-21T23:46:48.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.224+0000 7fdf3d9c0700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3819ebb0 con 0x7fdf38101190 2024-02-21T23:46:48.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.224+0000 7fdf36ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf380ffec0 0x7fdf38196de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:48.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.224+0000 7fdf36ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf380ffec0 0x7fdf38196de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49490/0 (socket says 172.21.15.90:49490) 2024-02-21T23:46:48.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.224+0000 7fdf36ffd700 1 -- 172.21.15.90:0/3121205601 learned_addr learned my addr 172.21.15.90:0/3121205601 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:48.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.224+0000 7fdf377fe700 1 --2- 172.21.15.90:0/3121205601 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdf38101190 0x7fdf3819e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:48.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.224+0000 7fdf367fc700 1 --2- 172.21.15.90:0/3121205601 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf38100860 0x7fdf38197320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:48.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.224+0000 7fdf36ffd700 1 -- 172.21.15.90:0/3121205601 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdf38101190 msgr2=0x7fdf3819e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:48.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.224+0000 7fdf36ffd700 1 --2- 172.21.15.90:0/3121205601 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdf38101190 0x7fdf3819e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:48.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.224+0000 7fdf36ffd700 1 -- 172.21.15.90:0/3121205601 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf38100860 msgr2=0x7fdf38197320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:48.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.224+0000 7fdf36ffd700 1 --2- 172.21.15.90:0/3121205601 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf38100860 0x7fdf38197320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:48.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.224+0000 7fdf36ffd700 1 -- 172.21.15.90:0/3121205601 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf30009d00 con 0x7fdf380ffec0 2024-02-21T23:46:48.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.224+0000 7fdf367fc700 1 --2- 172.21.15.90:0/3121205601 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf38100860 0x7fdf38197320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:46:48.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.224+0000 7fdf36ffd700 1 --2- 172.21.15.90:0/3121205601 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf380ffec0 0x7fdf38196de0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c00c990 tx=0x7fdf2c00cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:48.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.224+0000 7fdf3c9be700 1 -- 172.21.15.90:0/3121205601 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf2c007b00 con 0x7fdf380ffec0 2024-02-21T23:46:48.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.224+0000 7fdf3c9be700 1 -- 172.21.15.90:0/3121205601 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdf2c007c60 con 0x7fdf380ffec0 2024-02-21T23:46:48.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.224+0000 7fdf3c9be700 1 -- 172.21.15.90:0/3121205601 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf2c018660 con 0x7fdf380ffec0 2024-02-21T23:46:48.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.224+0000 7fdf3d9c0700 1 -- 172.21.15.90:0/3121205601 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf3819ee90 con 0x7fdf380ffec0 2024-02-21T23:46:48.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.224+0000 7fdf3d9c0700 1 -- 172.21.15.90:0/3121205601 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf3819f260 con 0x7fdf380ffec0 2024-02-21T23:46:48.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.225+0000 7fdf3d9c0700 1 -- 172.21.15.90:0/3121205601 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf380671f0 con 0x7fdf380ffec0 2024-02-21T23:46:48.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.226+0000 7fdf3c9be700 1 -- 172.21.15.90:0/3121205601 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7fdf2c00f450 con 0x7fdf380ffec0 2024-02-21T23:46:48.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.227+0000 7fdf3c9be700 1 --2- 172.21.15.90:0/3121205601 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fdf200a1120 0x7fdf200a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:48.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.227+0000 7fdf3c9be700 1 -- 172.21.15.90:0/3121205601 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fdf2c0cc9f0 con 0x7fdf380ffec0 2024-02-21T23:46:48.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.227+0000 7fdf367fc700 1 --2- 172.21.15.90:0/3121205601 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fdf200a1120 0x7fdf200a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:48.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.227+0000 7fdf367fc700 1 --2- 172.21.15.90:0/3121205601 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fdf200a1120 0x7fdf200a35e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fdf28005950 tx=0x7fdf280058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:48.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.231+0000 7fdf3c9be700 1 -- 172.21.15.90:0/3121205601 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fdf2c095f90 con 0x7fdf380ffec0 2024-02-21T23:46:48.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.428+0000 7fdf3d9c0700 1 -- 172.21.15.90:0/3121205601 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdf3819f820 con 0x7fdf380ffec0 2024-02-21T23:46:48.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.428+0000 7fdf3c9be700 1 -- 172.21.15.90:0/3121205601 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdf2c0995b0 con 0x7fdf380ffec0 2024-02-21T23:46:48.413 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-02-21T23:46:48.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.430+0000 7fdf3d9c0700 1 -- 172.21.15.90:0/3121205601 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fdf200a1120 msgr2=0x7fdf200a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:48.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.430+0000 7fdf3d9c0700 1 --2- 172.21.15.90:0/3121205601 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fdf200a1120 0x7fdf200a35e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fdf28005950 tx=0x7fdf280058e0 comp rx=0 tx=0).stop 2024-02-21T23:46:48.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.430+0000 7fdf3d9c0700 1 -- 172.21.15.90:0/3121205601 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf380ffec0 msgr2=0x7fdf38196de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:48.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.430+0000 7fdf3d9c0700 1 --2- 172.21.15.90:0/3121205601 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf380ffec0 0x7fdf38196de0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c00c990 tx=0x7fdf2c00cca0 comp rx=0 tx=0).stop 2024-02-21T23:46:48.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.430+0000 7fdf3d9c0700 1 -- 172.21.15.90:0/3121205601 shutdown_connections 2024-02-21T23:46:48.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.430+0000 7fdf3d9c0700 1 --2- 172.21.15.90:0/3121205601 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fdf200a1120 0x7fdf200a35e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:48.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.430+0000 7fdf3d9c0700 1 --2- 172.21.15.90:0/3121205601 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdf38101190 0x7fdf3819e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:48.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.430+0000 7fdf3d9c0700 1 --2- 172.21.15.90:0/3121205601 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf380ffec0 0x7fdf38196de0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:48.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.430+0000 7fdf3d9c0700 1 --2- 172.21.15.90:0/3121205601 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf38100860 0x7fdf38197320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:48.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.430+0000 7fdf3d9c0700 1 -- 172.21.15.90:0/3121205601 >> 172.21.15.90:0/3121205601 conn(0x7fdf380fa8f0 msgr2=0x7fdf380fc380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:48.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.430+0000 7fdf3d9c0700 1 -- 172.21.15.90:0/3121205601 shutdown_connections 2024-02-21T23:46:48.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.430+0000 7fdf3d9c0700 1 -- 172.21.15.90:0/3121205601 wait complete. 2024-02-21T23:46:48.427 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-21T23:46:48.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.569+0000 7f7721456700 1 -- 172.21.15.90:0/3167011754 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f771c0ff3a0 msgr2=0x7f771c0ff7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:48.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.569+0000 7f7721456700 1 --2- 172.21.15.90:0/3167011754 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f771c0ff3a0 0x7f771c0ff7c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f77100099c0 tx=0x7f7710009cd0 comp rx=0 tx=0).stop 2024-02-21T23:46:48.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.569+0000 7f7721456700 1 -- 172.21.15.90:0/3167011754 shutdown_connections 2024-02-21T23:46:48.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.569+0000 7f7721456700 1 --2- 172.21.15.90:0/3167011754 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f771c100f30 0x7f771c1077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:48.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.569+0000 7f7721456700 1 --2- 172.21.15.90:0/3167011754 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f771c0ff3a0 0x7f771c0ff7c0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:48.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.569+0000 7f7721456700 1 --2- 172.21.15.90:0/3167011754 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f771c100590 0x7f771c1009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:48.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.569+0000 7f7721456700 1 -- 172.21.15.90:0/3167011754 >> 172.21.15.90:0/3167011754 conn(0x7f771c0fa910 msgr2=0x7f771c0fcd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:48.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.569+0000 7f7721456700 1 -- 172.21.15.90:0/3167011754 shutdown_connections 2024-02-21T23:46:48.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.569+0000 7f7721456700 1 -- 172.21.15.90:0/3167011754 wait complete. 2024-02-21T23:46:48.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.569+0000 7f7721456700 1 Processor -- start 2024-02-21T23:46:48.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.570+0000 7f7721456700 1 -- start start 2024-02-21T23:46:48.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.570+0000 7f7721456700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f771c0ff3a0 0x7f771c196df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:48.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.570+0000 7f7721456700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f771c100590 0x7f771c197330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:48.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.570+0000 7f7721456700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f771c100f30 0x7f771c19e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:48.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.570+0000 7f7721456700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771c19e8d0 con 0x7f771c100f30 2024-02-21T23:46:48.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.570+0000 7f7721456700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771c19ea10 con 0x7f771c100590 2024-02-21T23:46:48.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.570+0000 7f7721456700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771c19eba0 con 0x7f771c0ff3a0 2024-02-21T23:46:48.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.570+0000 7f771b7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f771c100f30 0x7f771c19e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:48.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.570+0000 7f771b7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f771c100f30 0x7f771c19e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59524/0 (socket says 172.21.15.90:59524) 2024-02-21T23:46:48.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.570+0000 7f771b7fe700 1 -- 172.21.15.90:0/1475197057 learned_addr learned my addr 172.21.15.90:0/1475197057 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:48.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.570+0000 7f771affd700 1 --2- 172.21.15.90:0/1475197057 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f771c0ff3a0 0x7f771c196df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:48.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.570+0000 7f771a7fc700 1 --2- 172.21.15.90:0/1475197057 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f771c100590 0x7f771c197330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:48.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.570+0000 7f771b7fe700 1 -- 172.21.15.90:0/1475197057 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f771c0ff3a0 msgr2=0x7f771c196df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:48.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.570+0000 7f771b7fe700 1 --2- 172.21.15.90:0/1475197057 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f771c0ff3a0 0x7f771c196df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:48.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.570+0000 7f771b7fe700 1 -- 172.21.15.90:0/1475197057 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f771c100590 msgr2=0x7f771c197330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:48.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.570+0000 7f771b7fe700 1 --2- 172.21.15.90:0/1475197057 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f771c100590 0x7f771c197330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:48.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.570+0000 7f771b7fe700 1 -- 172.21.15.90:0/1475197057 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77100095f0 con 0x7f771c100f30 2024-02-21T23:46:48.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.570+0000 7f771a7fc700 1 --2- 172.21.15.90:0/1475197057 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f771c100590 0x7f771c197330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:46:48.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.570+0000 7f771affd700 1 --2- 172.21.15.90:0/1475197057 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f771c0ff3a0 0x7f771c196df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:46:48.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.570+0000 7f771b7fe700 1 --2- 172.21.15.90:0/1475197057 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f771c100f30 0x7f771c19e300 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f771400ada0 tx=0x7f771400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:48.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.571+0000 7f7703fff700 1 -- 172.21.15.90:0/1475197057 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771400f070 con 0x7f771c100f30 2024-02-21T23:46:48.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.571+0000 7f7703fff700 1 -- 172.21.15.90:0/1475197057 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7714007b30 con 0x7f771c100f30 2024-02-21T23:46:48.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.571+0000 7f7721456700 1 -- 172.21.15.90:0/1475197057 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f771c19ee80 con 0x7f771c100f30 2024-02-21T23:46:48.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.571+0000 7f7703fff700 1 -- 172.21.15.90:0/1475197057 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7714005440 con 0x7f771c100f30 2024-02-21T23:46:48.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.571+0000 7f7721456700 1 -- 172.21.15.90:0/1475197057 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f771c19f2b0 con 0x7f771c100f30 2024-02-21T23:46:48.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.572+0000 7f7721456700 1 -- 172.21.15.90:0/1475197057 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f771c067ed0 con 0x7f771c100f30 2024-02-21T23:46:48.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.573+0000 7f7703fff700 1 -- 172.21.15.90:0/1475197057 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7f7714004ad0 con 0x7f771c100f30 2024-02-21T23:46:48.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.573+0000 7f7703fff700 1 --2- 172.21.15.90:0/1475197057 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f77040a1120 0x7f77040a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:48.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.573+0000 7f771affd700 1 --2- 172.21.15.90:0/1475197057 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f77040a1120 0x7f77040a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:48.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.573+0000 7f7703fff700 1 -- 172.21.15.90:0/1475197057 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f77140cc1b0 con 0x7f771c100f30 2024-02-21T23:46:48.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.574+0000 7f771affd700 1 --2- 172.21.15.90:0/1475197057 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f77040a1120 0x7f77040a35e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f771000b580 tx=0x7f7710004500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:48.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.577+0000 7f7703fff700 1 -- 172.21.15.90:0/1475197057 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f77140957e0 con 0x7f771c100f30 2024-02-21T23:46:48.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:48 smithi090 ceph-mon[43385]: pgmap v232: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 342 B/s wr, 0 op/s 2024-02-21T23:46:48.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:48 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/3515830779' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-21T23:46:48.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:48 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/4180373811' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-21T23:46:48.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:48 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/3121205601' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-02-21T23:46:48.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.772+0000 7f7721456700 1 -- 172.21.15.90:0/1475197057 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f771c19f590 con 0x7f771c100f30 2024-02-21T23:46:48.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.772+0000 7f7703fff700 1 -- 172.21.15.90:0/1475197057 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7714098e00 con 0x7f771c100f30 2024-02-21T23:46:48.757 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-02-21T23:46:48.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.775+0000 7f7721456700 1 -- 172.21.15.90:0/1475197057 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f77040a1120 msgr2=0x7f77040a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:48.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.775+0000 7f7721456700 1 --2- 172.21.15.90:0/1475197057 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f77040a1120 0x7f77040a35e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f771000b580 tx=0x7f7710004500 comp rx=0 tx=0).stop 2024-02-21T23:46:48.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.775+0000 7f7721456700 1 -- 172.21.15.90:0/1475197057 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f771c100f30 msgr2=0x7f771c19e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:48.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.775+0000 7f7721456700 1 --2- 172.21.15.90:0/1475197057 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f771c100f30 0x7f771c19e300 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f771400ada0 tx=0x7f771400c330 comp rx=0 tx=0).stop 2024-02-21T23:46:48.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.775+0000 7f7721456700 1 -- 172.21.15.90:0/1475197057 shutdown_connections 2024-02-21T23:46:48.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.775+0000 7f7721456700 1 --2- 172.21.15.90:0/1475197057 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f77040a1120 0x7f77040a35e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:48.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.775+0000 7f7721456700 1 --2- 172.21.15.90:0/1475197057 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f771c100f30 0x7f771c19e300 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:48.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.775+0000 7f7721456700 1 --2- 172.21.15.90:0/1475197057 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f771c0ff3a0 0x7f771c196df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:48.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.775+0000 7f7721456700 1 --2- 172.21.15.90:0/1475197057 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f771c100590 0x7f771c197330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:48.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.775+0000 7f7721456700 1 -- 172.21.15.90:0/1475197057 >> 172.21.15.90:0/1475197057 conn(0x7f771c0fa910 msgr2=0x7f771c0fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:48.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.775+0000 7f7721456700 1 -- 172.21.15.90:0/1475197057 shutdown_connections 2024-02-21T23:46:48.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.775+0000 7f7721456700 1 -- 172.21.15.90:0/1475197057 wait complete. 2024-02-21T23:46:48.772 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-02-21T23:46:48.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:48 smithi176 ceph-mon[47172]: pgmap v232: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 342 B/s wr, 0 op/s 2024-02-21T23:46:48.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:48 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/3515830779' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-21T23:46:48.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:48 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/4180373811' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-21T23:46:48.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:48 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/3121205601' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-02-21T23:46:48.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.912+0000 7fd6f53eb700 1 -- 172.21.15.90:0/462270871 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6f0101770 msgr2=0x7fd6f0103b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:48.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.912+0000 7fd6f53eb700 1 --2- 172.21.15.90:0/462270871 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6f0101770 0x7fd6f0103b60 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4009990 tx=0x7fd6e4009ca0 comp rx=0 tx=0).stop 2024-02-21T23:46:48.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.912+0000 7fd6f53eb700 1 -- 172.21.15.90:0/462270871 shutdown_connections 2024-02-21T23:46:48.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.912+0000 7fd6f53eb700 1 --2- 172.21.15.90:0/462270871 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6f01069d0 0x7fd6f0108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:48.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.912+0000 7fd6f53eb700 1 --2- 172.21.15.90:0/462270871 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6f0101770 0x7fd6f0103b60 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:48.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.912+0000 7fd6f53eb700 1 --2- 172.21.15.90:0/462270871 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd6f01040a0 0x7fd6f0106490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:48.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.912+0000 7fd6f53eb700 1 -- 172.21.15.90:0/462270871 >> 172.21.15.90:0/462270871 conn(0x7fd6f00fb160 msgr2=0x7fd6f00fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:48.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.912+0000 7fd6f53eb700 1 -- 172.21.15.90:0/462270871 shutdown_connections 2024-02-21T23:46:48.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.912+0000 7fd6f53eb700 1 -- 172.21.15.90:0/462270871 wait complete. 2024-02-21T23:46:48.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.912+0000 7fd6f53eb700 1 Processor -- start 2024-02-21T23:46:48.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.913+0000 7fd6f53eb700 1 -- start start 2024-02-21T23:46:48.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.913+0000 7fd6f53eb700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6f0101770 0x7fd6f0196c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:48.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.913+0000 7fd6f53eb700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd6f01040a0 0x7fd6f0197140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:48.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.913+0000 7fd6f53eb700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6f01069d0 0x7fd6f019e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:48.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.913+0000 7fd6f53eb700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f019e730 con 0x7fd6f0101770 2024-02-21T23:46:48.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.913+0000 7fd6f53eb700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f019e870 con 0x7fd6f01040a0 2024-02-21T23:46:48.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.913+0000 7fd6f53eb700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f019ea00 con 0x7fd6f01069d0 2024-02-21T23:46:48.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.913+0000 7fd6eeffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6f0101770 0x7fd6f0196c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:48.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.913+0000 7fd6eeffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6f0101770 0x7fd6f0196c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59542/0 (socket says 172.21.15.90:59542) 2024-02-21T23:46:48.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.913+0000 7fd6eeffd700 1 -- 172.21.15.90:0/2163937641 learned_addr learned my addr 172.21.15.90:0/2163937641 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:48.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.913+0000 7fd6ef7fe700 1 --2- 172.21.15.90:0/2163937641 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6f01069d0 0x7fd6f019e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:48.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.913+0000 7fd6ee7fc700 1 --2- 172.21.15.90:0/2163937641 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd6f01040a0 0x7fd6f0197140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:48.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.913+0000 7fd6eeffd700 1 -- 172.21.15.90:0/2163937641 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6f01069d0 msgr2=0x7fd6f019e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:48.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.913+0000 7fd6eeffd700 1 --2- 172.21.15.90:0/2163937641 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6f01069d0 0x7fd6f019e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:48.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.913+0000 7fd6eeffd700 1 -- 172.21.15.90:0/2163937641 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd6f01040a0 msgr2=0x7fd6f0197140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:48.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.913+0000 7fd6eeffd700 1 --2- 172.21.15.90:0/2163937641 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd6f01040a0 0x7fd6f0197140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:48.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.913+0000 7fd6eeffd700 1 -- 172.21.15.90:0/2163937641 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e40095f0 con 0x7fd6f0101770 2024-02-21T23:46:48.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.913+0000 7fd6ee7fc700 1 --2- 172.21.15.90:0/2163937641 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd6f01040a0 0x7fd6f0197140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:46:48.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.913+0000 7fd6eeffd700 1 --2- 172.21.15.90:0/2163937641 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6f0101770 0x7fd6f0196c00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4000c00 tx=0x7fd6e4005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:48.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.913+0000 7fd6d7fff700 1 -- 172.21.15.90:0/2163937641 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e401d070 con 0x7fd6f0101770 2024-02-21T23:46:48.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.914+0000 7fd6f53eb700 1 -- 172.21.15.90:0/2163937641 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6f019ec80 con 0x7fd6f0101770 2024-02-21T23:46:48.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.914+0000 7fd6d7fff700 1 -- 172.21.15.90:0/2163937641 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd6e4004ca0 con 0x7fd6f0101770 2024-02-21T23:46:48.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.914+0000 7fd6d7fff700 1 -- 172.21.15.90:0/2163937641 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e40055f0 con 0x7fd6f0101770 2024-02-21T23:46:48.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.914+0000 7fd6f53eb700 1 -- 172.21.15.90:0/2163937641 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6f019f0a0 con 0x7fd6f0101770 2024-02-21T23:46:48.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.914+0000 7fd6f53eb700 1 -- 172.21.15.90:0/2163937641 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6f00fcce0 con 0x7fd6f0101770 2024-02-21T23:46:48.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.916+0000 7fd6d7fff700 1 -- 172.21.15.90:0/2163937641 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7fd6e400fc40 con 0x7fd6f0101770 2024-02-21T23:46:48.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.916+0000 7fd6d7fff700 1 --2- 172.21.15.90:0/2163937641 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fd6d80a1120 0x7fd6d80a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:48.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.916+0000 7fd6ee7fc700 1 --2- 172.21.15.90:0/2163937641 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fd6d80a1120 0x7fd6d80a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:48.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.916+0000 7fd6d7fff700 1 -- 172.21.15.90:0/2163937641 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fd6e40ce010 con 0x7fd6f0101770 2024-02-21T23:46:48.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.917+0000 7fd6ee7fc700 1 --2- 172.21.15.90:0/2163937641 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fd6d80a1120 0x7fd6d80a35e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0005fd0 tx=0x7fd6e0005ee0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:48.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:48.920+0000 7fd6d7fff700 1 -- 172.21.15.90:0/2163937641 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fd6e4097640 con 0x7fd6f0101770 2024-02-21T23:46:48.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:48 smithi178 ceph-mon[47547]: pgmap v232: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 342 B/s wr, 0 op/s 2024-02-21T23:46:48.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:48 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/3515830779' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-21T23:46:48.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:48 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/4180373811' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-21T23:46:48.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:48 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/3121205601' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-02-21T23:46:49.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.113+0000 7fd6f53eb700 1 -- 172.21.15.90:0/2163937641 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fd6f002cc70 con 0x7fd6f0101770 2024-02-21T23:46:49.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.114+0000 7fd6d7fff700 1 -- 172.21.15.90:0/2163937641 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v85) v1 ==== 103+0+11 (secure 0 0 0) 0x7fd6e409ac60 con 0x7fd6f0101770 2024-02-21T23:46:49.100 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-02-21T23:46:49.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.116+0000 7fd6f53eb700 1 -- 172.21.15.90:0/2163937641 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fd6d80a1120 msgr2=0x7fd6d80a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:49.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.116+0000 7fd6f53eb700 1 --2- 172.21.15.90:0/2163937641 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fd6d80a1120 0x7fd6d80a35e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0005fd0 tx=0x7fd6e0005ee0 comp rx=0 tx=0).stop 2024-02-21T23:46:49.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.116+0000 7fd6f53eb700 1 -- 172.21.15.90:0/2163937641 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6f0101770 msgr2=0x7fd6f0196c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:49.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.116+0000 7fd6f53eb700 1 --2- 172.21.15.90:0/2163937641 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6f0101770 0x7fd6f0196c00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4000c00 tx=0x7fd6e4005ef0 comp rx=0 tx=0).stop 2024-02-21T23:46:49.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.116+0000 7fd6f53eb700 1 -- 172.21.15.90:0/2163937641 shutdown_connections 2024-02-21T23:46:49.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.116+0000 7fd6f53eb700 1 --2- 172.21.15.90:0/2163937641 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fd6d80a1120 0x7fd6d80a35e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:49.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.116+0000 7fd6f53eb700 1 --2- 172.21.15.90:0/2163937641 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6f01069d0 0x7fd6f019e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:49.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.116+0000 7fd6f53eb700 1 --2- 172.21.15.90:0/2163937641 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6f0101770 0x7fd6f0196c00 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:49.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.116+0000 7fd6f53eb700 1 --2- 172.21.15.90:0/2163937641 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd6f01040a0 0x7fd6f0197140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:49.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.116+0000 7fd6f53eb700 1 -- 172.21.15.90:0/2163937641 >> 172.21.15.90:0/2163937641 conn(0x7fd6f00fb160 msgr2=0x7fd6f0103620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:49.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.116+0000 7fd6f53eb700 1 -- 172.21.15.90:0/2163937641 shutdown_connections 2024-02-21T23:46:49.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.116+0000 7fd6f53eb700 1 -- 172.21.15.90:0/2163937641 wait complete. 2024-02-21T23:46:49.113 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-02-21T23:46:49.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.250+0000 7fc65d1c1700 1 -- 172.21.15.90:0/2698522681 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6580ff730 msgr2=0x7fc6581092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:49.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.250+0000 7fc65d1c1700 1 --2- 172.21.15.90:0/2698522681 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6580ff730 0x7fc6581092e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc65000b0a0 tx=0x7fc65000b3b0 comp rx=0 tx=0).stop 2024-02-21T23:46:49.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.250+0000 7fc65d1c1700 1 -- 172.21.15.90:0/2698522681 shutdown_connections 2024-02-21T23:46:49.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.250+0000 7fc65d1c1700 1 --2- 172.21.15.90:0/2698522681 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6580ff730 0x7fc6581092e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:49.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.250+0000 7fc65d1c1700 1 --2- 172.21.15.90:0/2698522681 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc658102300 0x7fc658102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:49.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.250+0000 7fc65d1c1700 1 --2- 172.21.15.90:0/2698522681 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc6580fed70 0x7fc6580ff1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:49.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.250+0000 7fc65d1c1700 1 -- 172.21.15.90:0/2698522681 >> 172.21.15.90:0/2698522681 conn(0x7fc6580fa8f0 msgr2=0x7fc6580fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:49.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.250+0000 7fc65d1c1700 1 -- 172.21.15.90:0/2698522681 shutdown_connections 2024-02-21T23:46:49.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.250+0000 7fc65d1c1700 1 -- 172.21.15.90:0/2698522681 wait complete. 2024-02-21T23:46:49.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.251+0000 7fc65d1c1700 1 Processor -- start 2024-02-21T23:46:49.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.251+0000 7fc65d1c1700 1 -- start start 2024-02-21T23:46:49.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.251+0000 7fc65d1c1700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc6580fed70 0x7fc6581019b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:49.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.251+0000 7fc65d1c1700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6580ff730 0x7fc658100040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:49.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.251+0000 7fc65d1c1700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc658102300 0x7fc658100580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:49.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.251+0000 7fc65d1c1700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc658100d20 con 0x7fc6580ff730 2024-02-21T23:46:49.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.251+0000 7fc65d1c1700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc658100e90 con 0x7fc6580fed70 2024-02-21T23:46:49.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.251+0000 7fc65d1c1700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6581a0db0 con 0x7fc658102300 2024-02-21T23:46:49.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.251+0000 7fc65659c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6580ff730 0x7fc658100040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:49.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.251+0000 7fc65659c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6580ff730 0x7fc658100040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59556/0 (socket says 172.21.15.90:59556) 2024-02-21T23:46:49.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.251+0000 7fc65659c700 1 -- 172.21.15.90:0/2682497770 learned_addr learned my addr 172.21.15.90:0/2682497770 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:49.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.251+0000 7fc65759e700 1 --2- 172.21.15.90:0/2682497770 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc658102300 0x7fc658100580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:49.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.251+0000 7fc656d9d700 1 --2- 172.21.15.90:0/2682497770 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc6580fed70 0x7fc6581019b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:49.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.252+0000 7fc65659c700 1 -- 172.21.15.90:0/2682497770 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc658102300 msgr2=0x7fc658100580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:49.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.252+0000 7fc65659c700 1 --2- 172.21.15.90:0/2682497770 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc658102300 0x7fc658100580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:49.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.252+0000 7fc65659c700 1 -- 172.21.15.90:0/2682497770 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc6580fed70 msgr2=0x7fc6581019b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:49.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.252+0000 7fc65659c700 1 --2- 172.21.15.90:0/2682497770 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc6580fed70 0x7fc6581019b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:49.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.252+0000 7fc65659c700 1 -- 172.21.15.90:0/2682497770 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc650009d00 con 0x7fc6580ff730 2024-02-21T23:46:49.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.252+0000 7fc65659c700 1 --2- 172.21.15.90:0/2682497770 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6580ff730 0x7fc658100040 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc64800d980 tx=0x7fc64800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:49.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.252+0000 7fc64ffff700 1 -- 172.21.15.90:0/2682497770 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc648004020 con 0x7fc6580ff730 2024-02-21T23:46:49.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.252+0000 7fc65d1c1700 1 -- 172.21.15.90:0/2682497770 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6581a0fe0 con 0x7fc6580ff730 2024-02-21T23:46:49.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.252+0000 7fc64ffff700 1 -- 172.21.15.90:0/2682497770 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc64800f690 con 0x7fc6580ff730 2024-02-21T23:46:49.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.252+0000 7fc64ffff700 1 -- 172.21.15.90:0/2682497770 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc648003a60 con 0x7fc6580ff730 2024-02-21T23:46:49.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.252+0000 7fc65d1c1700 1 -- 172.21.15.90:0/2682497770 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6581a14c0 con 0x7fc6580ff730 2024-02-21T23:46:49.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.253+0000 7fc65d1c1700 1 -- 172.21.15.90:0/2682497770 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc658108250 con 0x7fc6580ff730 2024-02-21T23:46:49.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.254+0000 7fc64ffff700 1 -- 172.21.15.90:0/2682497770 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7fc64800fb00 con 0x7fc6580ff730 2024-02-21T23:46:49.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.254+0000 7fc64ffff700 1 --2- 172.21.15.90:0/2682497770 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fc6440a1120 0x7fc6440a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:49.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.255+0000 7fc656d9d700 1 --2- 172.21.15.90:0/2682497770 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fc6440a1120 0x7fc6440a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:49.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.255+0000 7fc64ffff700 1 -- 172.21.15.90:0/2682497770 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fc6480cc990 con 0x7fc6580ff730 2024-02-21T23:46:49.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.255+0000 7fc656d9d700 1 --2- 172.21.15.90:0/2682497770 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fc6440a1120 0x7fc6440a35e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc658067a80 tx=0x7fc640009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:49.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.258+0000 7fc64ffff700 1 -- 172.21.15.90:0/2682497770 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fc648095f30 con 0x7fc6580ff730 2024-02-21T23:46:49.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.474+0000 7fc65d1c1700 1 -- 172.21.15.90:0/2682497770 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc6580671f0 con 0x7fc6580ff730 2024-02-21T23:46:49.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.474+0000 7fc64ffff700 1 -- 172.21.15.90:0/2682497770 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v85) v1 ==== 81+0+591 (secure 0 0 0) 0x7fc648017260 con 0x7fc6580ff730 2024-02-21T23:46:49.460 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/159655312 2024-02-22T23:41:33.360021+0000 2024-02-21T23:46:49.460 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6801/1385551604 2024-02-22T23:41:33.360021+0000 2024-02-21T23:46:49.460 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/1452574770 2024-02-22T23:41:33.360021+0000 2024-02-21T23:46:49.460 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/3815292343 2024-02-22T23:41:11.484145+0000 2024-02-21T23:46:49.460 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6800/1385551604 2024-02-22T23:41:33.360021+0000 2024-02-21T23:46:49.460 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6801/4247695574 2024-02-22T23:41:11.484145+0000 2024-02-21T23:46:49.460 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/1328266622 2024-02-22T23:41:11.484145+0000 2024-02-21T23:46:49.460 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6800/4247695574 2024-02-22T23:41:11.484145+0000 2024-02-21T23:46:49.460 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/2569421522 2024-02-22T23:41:11.484145+0000 2024-02-21T23:46:49.460 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/3019701027 2024-02-22T23:41:33.360021+0000 2024-02-21T23:46:49.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.477+0000 7fc65d1c1700 1 -- 172.21.15.90:0/2682497770 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fc6440a1120 msgr2=0x7fc6440a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:49.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.477+0000 7fc65d1c1700 1 --2- 172.21.15.90:0/2682497770 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fc6440a1120 0x7fc6440a35e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc658067a80 tx=0x7fc640009260 comp rx=0 tx=0).stop 2024-02-21T23:46:49.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.477+0000 7fc65d1c1700 1 -- 172.21.15.90:0/2682497770 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6580ff730 msgr2=0x7fc658100040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:49.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.477+0000 7fc65d1c1700 1 --2- 172.21.15.90:0/2682497770 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6580ff730 0x7fc658100040 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc64800d980 tx=0x7fc64800dc90 comp rx=0 tx=0).stop 2024-02-21T23:46:49.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.477+0000 7fc65d1c1700 1 -- 172.21.15.90:0/2682497770 shutdown_connections 2024-02-21T23:46:49.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.477+0000 7fc65d1c1700 1 --2- 172.21.15.90:0/2682497770 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fc6440a1120 0x7fc6440a35e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:49.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.477+0000 7fc65d1c1700 1 --2- 172.21.15.90:0/2682497770 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc658102300 0x7fc658100580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:49.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.477+0000 7fc65d1c1700 1 --2- 172.21.15.90:0/2682497770 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc6580fed70 0x7fc6581019b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:49.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.477+0000 7fc65d1c1700 1 --2- 172.21.15.90:0/2682497770 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6580ff730 0x7fc658100040 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:49.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.477+0000 7fc65d1c1700 1 -- 172.21.15.90:0/2682497770 >> 172.21.15.90:0/2682497770 conn(0x7fc6580fa8f0 msgr2=0x7fc658105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:49.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.477+0000 7fc65d1c1700 1 -- 172.21.15.90:0/2682497770 shutdown_connections 2024-02-21T23:46:49.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:49.477+0000 7fc65d1c1700 1 -- 172.21.15.90:0/2682497770 wait complete. 2024-02-21T23:46:49.463 INFO:teuthology.orchestra.run.smithi090.stderr:listed 10 entries 2024-02-21T23:46:49.474 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi090.front.sepia.ceph.com... 2024-02-21T23:46:49.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:46:49.474 DEBUG:teuthology.orchestra.run.smithi090:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-02-21T23:46:49.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi090.stderr:ceph-fuse[97587]: fuse finished with error 0 and tester_r 0 2024-02-21T23:46:49.522 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:46:49.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:49 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/1475197057' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-02-21T23:46:49.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:49 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/2163937641' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-02-21T23:46:49.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:49 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/2682497770' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-21T23:46:49.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:49 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/1475197057' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-02-21T23:46:49.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:49 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/2163937641' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-02-21T23:46:49.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:49 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/2682497770' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-21T23:46:49.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:49 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/1475197057' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-02-21T23:46:49.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:49 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/2163937641' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-02-21T23:46:49.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:49 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/2682497770' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-21T23:46:50.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:50 smithi176 ceph-mon[47172]: pgmap v233: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 291 B/s wr, 0 op/s 2024-02-21T23:46:50.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:50 smithi178 ceph-mon[47547]: pgmap v233: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 291 B/s wr, 0 op/s 2024-02-21T23:46:50.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:50 smithi090 ceph-mon[43385]: pgmap v233: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 291 B/s wr, 0 op/s 2024-02-21T23:46:52.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:52 smithi176 ceph-mon[47172]: pgmap v234: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-02-21T23:46:52.924 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:52 smithi178 ceph-mon[47547]: pgmap v234: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-02-21T23:46:52.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:52 smithi090 ceph-mon[43385]: pgmap v234: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-02-21T23:46:54.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:54 smithi176 ceph-mon[47172]: pgmap v235: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-02-21T23:46:54.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:54 smithi178 ceph-mon[47547]: pgmap v235: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-02-21T23:46:54.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:54 smithi090 ceph-mon[43385]: pgmap v235: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-02-21T23:46:55.623 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi090.front.sepia.ceph.com 2024-02-21T23:46:55.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:46:55.624 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-02-21T23:46:55.658 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:46:55.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.846+0000 7f9a3dde5700 1 -- 172.21.15.90:0/2968403333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a38100ce0 msgr2=0x7f9a38101140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:55.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.846+0000 7f9a3dde5700 1 --2- 172.21.15.90:0/2968403333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a38100ce0 0x7f9a38101140 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c0099c0 tx=0x7f9a2c009cd0 comp rx=0 tx=0).stop 2024-02-21T23:46:55.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.846+0000 7f9a3dde5700 1 -- 172.21.15.90:0/2968403333 shutdown_connections 2024-02-21T23:46:55.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.846+0000 7f9a3dde5700 1 --2- 172.21.15.90:0/2968403333 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a38101680 0x7f9a38107f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:55.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.846+0000 7f9a3dde5700 1 --2- 172.21.15.90:0/2968403333 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9a380ffaf0 0x7f9a380fff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:55.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.846+0000 7f9a3dde5700 1 --2- 172.21.15.90:0/2968403333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a38100ce0 0x7f9a38101140 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:55.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.846+0000 7f9a3dde5700 1 -- 172.21.15.90:0/2968403333 >> 172.21.15.90:0/2968403333 conn(0x7f9a380fb060 msgr2=0x7f9a380fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:55.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.846+0000 7f9a3dde5700 1 -- 172.21.15.90:0/2968403333 shutdown_connections 2024-02-21T23:46:55.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.846+0000 7f9a3dde5700 1 -- 172.21.15.90:0/2968403333 wait complete. 2024-02-21T23:46:55.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.847+0000 7f9a3dde5700 1 Processor -- start 2024-02-21T23:46:55.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.847+0000 7f9a3dde5700 1 -- start start 2024-02-21T23:46:55.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.847+0000 7f9a3dde5700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a380ffaf0 0x7f9a38103a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:55.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.847+0000 7f9a3dde5700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9a38100ce0 0x7f9a381020c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:55.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.847+0000 7f9a3dde5700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a38101680 0x7f9a38102600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:55.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.847+0000 7f9a3dde5700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a38102da0 con 0x7f9a380ffaf0 2024-02-21T23:46:55.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.847+0000 7f9a3dde5700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a38102f10 con 0x7f9a38101680 2024-02-21T23:46:55.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.847+0000 7f9a3dde5700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a381a0b60 con 0x7f9a38100ce0 2024-02-21T23:46:55.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.847+0000 7f9a377fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a380ffaf0 0x7f9a38103a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:55.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.848+0000 7f9a377fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a380ffaf0 0x7f9a38103a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59568/0 (socket says 172.21.15.90:59568) 2024-02-21T23:46:55.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.848+0000 7f9a377fe700 1 -- 172.21.15.90:0/4005308737 learned_addr learned my addr 172.21.15.90:0/4005308737 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:55.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.848+0000 7f9a36ffd700 1 --2- 172.21.15.90:0/4005308737 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9a38100ce0 0x7f9a381020c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:55.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.848+0000 7f9a37fff700 1 --2- 172.21.15.90:0/4005308737 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a38101680 0x7f9a38102600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:55.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.848+0000 7f9a377fe700 1 -- 172.21.15.90:0/4005308737 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9a38100ce0 msgr2=0x7f9a381020c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:55.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.848+0000 7f9a377fe700 1 --2- 172.21.15.90:0/4005308737 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9a38100ce0 0x7f9a381020c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:55.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.848+0000 7f9a377fe700 1 -- 172.21.15.90:0/4005308737 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a38101680 msgr2=0x7f9a38102600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:55.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.848+0000 7f9a377fe700 1 --2- 172.21.15.90:0/4005308737 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a38101680 0x7f9a38102600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:55.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.848+0000 7f9a377fe700 1 -- 172.21.15.90:0/4005308737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a2c0095f0 con 0x7f9a380ffaf0 2024-02-21T23:46:55.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.848+0000 7f9a37fff700 1 --2- 172.21.15.90:0/4005308737 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a38101680 0x7f9a38102600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:46:55.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.848+0000 7f9a36ffd700 1 --2- 172.21.15.90:0/4005308737 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9a38100ce0 0x7f9a381020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:46:55.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.848+0000 7f9a377fe700 1 --2- 172.21.15.90:0/4005308737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a380ffaf0 0x7f9a38103a30 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9a2800d980 tx=0x7f9a2800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:55.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.848+0000 7f9a34ff9700 1 -- 172.21.15.90:0/4005308737 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a2800de60 con 0x7f9a380ffaf0 2024-02-21T23:46:55.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.848+0000 7f9a34ff9700 1 -- 172.21.15.90:0/4005308737 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9a280044e0 con 0x7f9a380ffaf0 2024-02-21T23:46:55.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.848+0000 7f9a3dde5700 1 -- 172.21.15.90:0/4005308737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a381a0df0 con 0x7f9a380ffaf0 2024-02-21T23:46:55.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.848+0000 7f9a34ff9700 1 -- 172.21.15.90:0/4005308737 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a28003b40 con 0x7f9a380ffaf0 2024-02-21T23:46:55.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.848+0000 7f9a3dde5700 1 -- 172.21.15.90:0/4005308737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a381a1210 con 0x7f9a380ffaf0 2024-02-21T23:46:55.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.849+0000 7f9a3dde5700 1 -- 172.21.15.90:0/4005308737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a380679a0 con 0x7f9a380ffaf0 2024-02-21T23:46:55.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.850+0000 7f9a34ff9700 1 -- 172.21.15.90:0/4005308737 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7f9a28004870 con 0x7f9a380ffaf0 2024-02-21T23:46:55.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.851+0000 7f9a34ff9700 1 --2- 172.21.15.90:0/4005308737 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f9a200a1120 0x7f9a200a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:55.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.851+0000 7f9a34ff9700 1 -- 172.21.15.90:0/4005308737 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f9a280ccc30 con 0x7f9a380ffaf0 2024-02-21T23:46:55.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.851+0000 7f9a36ffd700 1 --2- 172.21.15.90:0/4005308737 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f9a200a1120 0x7f9a200a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:55.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.851+0000 7f9a36ffd700 1 --2- 172.21.15.90:0/4005308737 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f9a200a1120 0x7f9a200a35e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c009fd0 tx=0x7f9a2c009530 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:55.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:55.854+0000 7f9a34ff9700 1 -- 172.21.15.90:0/4005308737 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f9a280961d0 con 0x7f9a380ffaf0 2024-02-21T23:46:56.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.056+0000 7f9a3dde5700 1 -- 172.21.15.90:0/4005308737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a381a1620 con 0x7f9a380ffaf0 2024-02-21T23:46:56.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.057+0000 7f9a34ff9700 1 -- 172.21.15.90:0/4005308737 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23925 (secure 0 0 0) 0x7f9a280997f0 con 0x7f9a380ffaf0 2024-02-21T23:46:56.043 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:46:56.043 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":85,"fsid":"7ed81218-d112-11ee-95bd-87774f69a715","created":"2024-02-21T23:40:41.785526+0000","modified":"2024-02-21T23:46:38.878698+0000","last_up_change":"2024-02-21T23:46:00.860501+0000","last_in_change":"2024-02-21T23:45:44.423815+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:43:21.675789+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:46:32.337047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-21T23:46:33.622440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-21T23:46:36.686123+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"77878890-d9a5-4d5b-a943-fec29ee0446e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6803","nonce":885319826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6805","nonce":885319826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6809","nonce":885319826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6807","nonce":885319826}]},"public_addr":"172.21.15.90:6803/885319826","cluster_addr":"172.21.15.90:6805/885319826","heartbeat_back_addr":"172.21.15.90:6809/885319826","heartbeat_front_addr":"172.21.15.90:6807/885319826","state":["exists","up"]},{"osd":1,"uuid":"7d6ea950-7a57-4f13-ab38-10a658912783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6801","nonce":4085240925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6803","nonce":4085240925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6807","nonce":4085240925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6805","nonce":4085240925}]},"public_addr":"172.21.15.176:6801/4085240925","cluster_addr":"172.21.15.176:6803/4085240925","heartbeat_back_addr":"172.21.15.176:6807/4085240925","heartbeat_front_addr":"172.21.15.176:6805/4085240925","state":["exists","up"]},{"osd":2,"uuid":"b83486bb-27f2-4d56-97fb-fb336fec4b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6801","nonce":3597250789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6803","nonce":3597250789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6807","nonce":3597250789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6805","nonce":3597250789}]},"public_addr":"172.21.15.178:6801/3597250789","cluster_addr":"172.21.15.178:6803/3597250789","heartbeat_back_addr":"172.21.15.178:6807/3597250789","heartbeat_front_addr":"172.21.15.178:6805/3597250789","state":["exists","up"]},{"osd":3,"uuid":"870e5bb6-eed5-4a0a-8e03-38de4f816b44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6811","nonce":3299043821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6813","nonce":3299043821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6817","nonce":3299043821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6815","nonce":3299043821}]},"public_addr":"172.21.15.90:6811/3299043821","cluster_addr":"172.21.15.90:6813/3299043821","heartbeat_back_addr":"172.21.15.90:6817/3299043821","heartbeat_front_addr":"172.21.15.90:6815/3299043821","state":["exists","up"]},{"osd":4,"uuid":"cb9f7f48-76e0-4368-ab87-2a09a8487ccd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6809","nonce":3548915687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6811","nonce":3548915687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6815","nonce":3548915687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6813","nonce":3548915687}]},"public_addr":"172.21.15.176:6809/3548915687","cluster_addr":"172.21.15.176:6811/3548915687","heartbeat_back_addr":"172.21.15.176:6815/3548915687","heartbeat_front_addr":"172.21.15.176:6813/3548915687","state":["exists","up"]},{"osd":5,"uuid":"54749022-c50d-4831-a1d6-8412b59c8a01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6809","nonce":286274518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6811","nonce":286274518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6815","nonce":286274518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6813","nonce":286274518}]},"public_addr":"172.21.15.178:6809/286274518","cluster_addr":"172.21.15.178:6811/286274518","heartbeat_back_addr":"172.21.15.178:6815/286274518","heartbeat_front_addr":"172.21.15.178:6813/286274518","state":["exists","up"]},{"osd":6,"uuid":"09bb5393-2efd-4a63-b0e8-99bef5bced13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6819","nonce":67640620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6821","nonce":67640620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6825","nonce":67640620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6823","nonce":67640620}]},"public_addr":"172.21.15.90:6819/67640620","cluster_addr":"172.21.15.90:6821/67640620","heartbeat_back_addr":"172.21.15.90:6825/67640620","heartbeat_front_addr":"172.21.15.90:6823/67640620","state":["exists","up"]},{"osd":7,"uuid":"7a859e69-6b63-467b-a3b3-99247d66f5c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6817","nonce":3998561598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6819","nonce":3998561598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6823","nonce":3998561598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6821","nonce":3998561598}]},"public_addr":"172.21.15.176:6817/3998561598","cluster_addr":"172.21.15.176:6819/3998561598","heartbeat_back_addr":"172.21.15.176:6823/3998561598","heartbeat_front_addr":"172.21.15.176:6821/3998561598","state":["exists","up"]},{"osd":8,"uuid":"bc23144e-22a5-4894-a8b5-9323383574d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6817","nonce":490229963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6819","nonce":490229963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6823","nonce":490229963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6821","nonce":490229963}]},"public_addr":"172.21.15.178:6817/490229963","cluster_addr":"172.21.15.178:6819/490229963","heartbeat_back_addr":"172.21.15.178:6823/490229963","heartbeat_front_addr":"172.21.15.178:6821/490229963","state":["exists","up"]},{"osd":9,"uuid":"0c2cecd4-55de-4fc4-a6f4-ca97c8460444","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6827","nonce":2061394761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6829","nonce":2061394761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6833","nonce":2061394761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6831","nonce":2061394761}]},"public_addr":"172.21.15.90:6827/2061394761","cluster_addr":"172.21.15.90:6829/2061394761","heartbeat_back_addr":"172.21.15.90:6833/2061394761","heartbeat_front_addr":"172.21.15.90:6831/2061394761","state":["exists","up"]},{"osd":10,"uuid":"fa04232b-b1ec-447a-8e94-7c9bf26d0291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6825","nonce":639411757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6827","nonce":639411757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6831","nonce":639411757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6829","nonce":639411757}]},"public_addr":"172.21.15.176:6825/639411757","cluster_addr":"172.21.15.176:6827/639411757","heartbeat_back_addr":"172.21.15.176:6831/639411757","heartbeat_front_addr":"172.21.15.176:6829/639411757","state":["exists","up"]},{"osd":11,"uuid":"7282e294-64fc-4856-9c74-86115f69408c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6825","nonce":4144536256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6827","nonce":4144536256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6831","nonce":4144536256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6829","nonce":4144536256}]},"public_addr":"172.21.15.178:6825/4144536256","cluster_addr":"172.21.15.178:6827/4144536256","heartbeat_back_addr":"172.21.15.178:6831/4144536256","heartbeat_front_addr":"172.21.15.178:6829/4144536256","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:42:44.280345+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:01.235630+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:19.408896+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:36.201749+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:53.401854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:44:11.342158+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:44:29.324502+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:44:46.821595+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:04.642344+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:23.019004+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:41.140149+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:59.401951+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/159655312":"2024-02-22T23:41:33.360021+0000","172.21.15.90:6801/1385551604":"2024-02-22T23:41:33.360021+0000","172.21.15.90:0/1452574770":"2024-02-22T23:41:33.360021+0000","172.21.15.90:0/3815292343":"2024-02-22T23:41:11.484145+0000","172.21.15.90:6800/1385551604":"2024-02-22T23:41:33.360021+0000","172.21.15.90:6801/4247695574":"2024-02-22T23:41:11.484145+0000","172.21.15.90:0/1328266622":"2024-02-22T23:41:11.484145+0000","172.21.15.90:6800/4247695574":"2024-02-22T23:41:11.484145+0000","172.21.15.90:0/2569421522":"2024-02-22T23:41:11.484145+0000","172.21.15.90:0/3019701027":"2024-02-22T23:41:33.360021+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:46:56.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.060+0000 7f9a3dde5700 1 -- 172.21.15.90:0/4005308737 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f9a200a1120 msgr2=0x7f9a200a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:56.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.060+0000 7f9a3dde5700 1 --2- 172.21.15.90:0/4005308737 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f9a200a1120 0x7f9a200a35e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c009fd0 tx=0x7f9a2c009530 comp rx=0 tx=0).stop 2024-02-21T23:46:56.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.060+0000 7f9a3dde5700 1 -- 172.21.15.90:0/4005308737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a380ffaf0 msgr2=0x7f9a38103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:56.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.060+0000 7f9a3dde5700 1 --2- 172.21.15.90:0/4005308737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a380ffaf0 0x7f9a38103a30 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9a2800d980 tx=0x7f9a2800dd40 comp rx=0 tx=0).stop 2024-02-21T23:46:56.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.060+0000 7f9a3dde5700 1 -- 172.21.15.90:0/4005308737 shutdown_connections 2024-02-21T23:46:56.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.060+0000 7f9a3dde5700 1 --2- 172.21.15.90:0/4005308737 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f9a200a1120 0x7f9a200a35e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:56.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.060+0000 7f9a3dde5700 1 --2- 172.21.15.90:0/4005308737 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a38101680 0x7f9a38102600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:56.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.060+0000 7f9a3dde5700 1 --2- 172.21.15.90:0/4005308737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a380ffaf0 0x7f9a38103a30 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:56.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.060+0000 7f9a3dde5700 1 --2- 172.21.15.90:0/4005308737 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9a38100ce0 0x7f9a381020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:56.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.060+0000 7f9a3dde5700 1 -- 172.21.15.90:0/4005308737 >> 172.21.15.90:0/4005308737 conn(0x7f9a380fb060 msgr2=0x7f9a380fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:56.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.060+0000 7f9a3dde5700 1 -- 172.21.15.90:0/4005308737 shutdown_connections 2024-02-21T23:46:56.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.060+0000 7f9a3dde5700 1 -- 172.21.15.90:0/4005308737 wait complete. 2024-02-21T23:46:56.058 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:46:56.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.194+0000 7fd301333700 1 -- 172.21.15.90:0/3026696642 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2fc101f70 msgr2=0x7fd2fc1023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:56.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.194+0000 7fd301333700 1 --2- 172.21.15.90:0/3026696642 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2fc101f70 0x7fd2fc1023d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec009990 tx=0x7fd2ec009ca0 comp rx=0 tx=0).stop 2024-02-21T23:46:56.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.194+0000 7fd301333700 1 -- 172.21.15.90:0/3026696642 shutdown_connections 2024-02-21T23:46:56.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.194+0000 7fd301333700 1 --2- 172.21.15.90:0/3026696642 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd2fc102910 0x7fd2fc1091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:56.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.194+0000 7fd301333700 1 --2- 172.21.15.90:0/3026696642 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2fc0fef00 0x7fd2fc0ff320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:56.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.194+0000 7fd301333700 1 --2- 172.21.15.90:0/3026696642 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2fc101f70 0x7fd2fc1023d0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:56.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.194+0000 7fd301333700 1 -- 172.21.15.90:0/3026696642 >> 172.21.15.90:0/3026696642 conn(0x7fd2fc0fa810 msgr2=0x7fd2fc0fcc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:56.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.194+0000 7fd301333700 1 -- 172.21.15.90:0/3026696642 shutdown_connections 2024-02-21T23:46:56.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.194+0000 7fd301333700 1 -- 172.21.15.90:0/3026696642 wait complete. 2024-02-21T23:46:56.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.194+0000 7fd301333700 1 Processor -- start 2024-02-21T23:46:56.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.194+0000 7fd301333700 1 -- start start 2024-02-21T23:46:56.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.195+0000 7fd301333700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2fc0fef00 0x7fd2fc196a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:56.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.195+0000 7fd301333700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd2fc101f70 0x7fd2fc196fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:56.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.195+0000 7fd301333700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2fc102910 0x7fd2fc19dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:56.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.195+0000 7fd301333700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc19e5b0 con 0x7fd2fc102910 2024-02-21T23:46:56.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.195+0000 7fd301333700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc19e6f0 con 0x7fd2fc0fef00 2024-02-21T23:46:56.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.195+0000 7fd301333700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc19e880 con 0x7fd2fc101f70 2024-02-21T23:46:56.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.195+0000 7fd2fb7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2fc102910 0x7fd2fc19dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:56.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.195+0000 7fd2fb7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2fc102910 0x7fd2fc19dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59580/0 (socket says 172.21.15.90:59580) 2024-02-21T23:46:56.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.195+0000 7fd2fb7fe700 1 -- 172.21.15.90:0/3316159251 learned_addr learned my addr 172.21.15.90:0/3316159251 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:56.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.195+0000 7fd2fa7fc700 1 --2- 172.21.15.90:0/3316159251 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd2fc101f70 0x7fd2fc196fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:56.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.195+0000 7fd2faffd700 1 --2- 172.21.15.90:0/3316159251 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2fc0fef00 0x7fd2fc196a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:56.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.195+0000 7fd2fb7fe700 1 -- 172.21.15.90:0/3316159251 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd2fc101f70 msgr2=0x7fd2fc196fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:56.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.195+0000 7fd2fb7fe700 1 --2- 172.21.15.90:0/3316159251 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd2fc101f70 0x7fd2fc196fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:56.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.195+0000 7fd2fb7fe700 1 -- 172.21.15.90:0/3316159251 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2fc0fef00 msgr2=0x7fd2fc196a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:56.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.195+0000 7fd2fb7fe700 1 --2- 172.21.15.90:0/3316159251 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2fc0fef00 0x7fd2fc196a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:56.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.195+0000 7fd2fb7fe700 1 -- 172.21.15.90:0/3316159251 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ec0095f0 con 0x7fd2fc102910 2024-02-21T23:46:56.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.195+0000 7fd2fa7fc700 1 --2- 172.21.15.90:0/3316159251 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd2fc101f70 0x7fd2fc196fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:46:56.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.195+0000 7fd2faffd700 1 --2- 172.21.15.90:0/3316159251 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2fc0fef00 0x7fd2fc196a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:46:56.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.195+0000 7fd2fb7fe700 1 --2- 172.21.15.90:0/3316159251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2fc102910 0x7fd2fc19dfe0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd2f4007d70 tx=0x7fd2f400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:56.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.195+0000 7fd2e3fff700 1 -- 172.21.15.90:0/3316159251 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f400ad40 con 0x7fd2fc102910 2024-02-21T23:46:56.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.195+0000 7fd301333700 1 -- 172.21.15.90:0/3316159251 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2fc19eb60 con 0x7fd2fc102910 2024-02-21T23:46:56.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.195+0000 7fd2e3fff700 1 -- 172.21.15.90:0/3316159251 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd2f40128e0 con 0x7fd2fc102910 2024-02-21T23:46:56.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.195+0000 7fd301333700 1 -- 172.21.15.90:0/3316159251 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2fc19f070 con 0x7fd2fc102910 2024-02-21T23:46:56.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.195+0000 7fd2e3fff700 1 -- 172.21.15.90:0/3316159251 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f4011760 con 0x7fd2fc102910 2024-02-21T23:46:56.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.196+0000 7fd301333700 1 -- 172.21.15.90:0/3316159251 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2fc108130 con 0x7fd2fc102910 2024-02-21T23:46:56.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.197+0000 7fd2e3fff700 1 -- 172.21.15.90:0/3316159251 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7fd2f4012a50 con 0x7fd2fc102910 2024-02-21T23:46:56.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.198+0000 7fd2e3fff700 1 --2- 172.21.15.90:0/3316159251 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fd2e40a1120 0x7fd2e40a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:56.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.198+0000 7fd2faffd700 1 --2- 172.21.15.90:0/3316159251 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fd2e40a1120 0x7fd2e40a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:56.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.198+0000 7fd2e3fff700 1 -- 172.21.15.90:0/3316159251 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fd2f40ccde0 con 0x7fd2fc102910 2024-02-21T23:46:56.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.198+0000 7fd2faffd700 1 --2- 172.21.15.90:0/3316159251 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fd2e40a1120 0x7fd2e40a35e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0005fd0 tx=0x7fd2f0005f00 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:56.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.202+0000 7fd2e3fff700 1 -- 172.21.15.90:0/3316159251 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fd2f4096380 con 0x7fd2fc102910 2024-02-21T23:46:56.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.405+0000 7fd301333700 1 -- 172.21.15.90:0/3316159251 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2fc067df0 con 0x7fd2fc102910 2024-02-21T23:46:56.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.407+0000 7fd2e3fff700 1 -- 172.21.15.90:0/3316159251 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23925 (secure 0 0 0) 0x7fd2f407fd90 con 0x7fd2fc102910 2024-02-21T23:46:56.392 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-21T23:46:56.392 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":85,"fsid":"7ed81218-d112-11ee-95bd-87774f69a715","created":"2024-02-21T23:40:41.785526+0000","modified":"2024-02-21T23:46:38.878698+0000","last_up_change":"2024-02-21T23:46:00.860501+0000","last_in_change":"2024-02-21T23:45:44.423815+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:43:21.675789+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:46:32.337047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-21T23:46:33.622440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-21T23:46:36.686123+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"77878890-d9a5-4d5b-a943-fec29ee0446e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6803","nonce":885319826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6805","nonce":885319826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6809","nonce":885319826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":885319826},{"type":"v1","addr":"172.21.15.90:6807","nonce":885319826}]},"public_addr":"172.21.15.90:6803/885319826","cluster_addr":"172.21.15.90:6805/885319826","heartbeat_back_addr":"172.21.15.90:6809/885319826","heartbeat_front_addr":"172.21.15.90:6807/885319826","state":["exists","up"]},{"osd":1,"uuid":"7d6ea950-7a57-4f13-ab38-10a658912783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6801","nonce":4085240925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6803","nonce":4085240925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6807","nonce":4085240925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":4085240925},{"type":"v1","addr":"172.21.15.176:6805","nonce":4085240925}]},"public_addr":"172.21.15.176:6801/4085240925","cluster_addr":"172.21.15.176:6803/4085240925","heartbeat_back_addr":"172.21.15.176:6807/4085240925","heartbeat_front_addr":"172.21.15.176:6805/4085240925","state":["exists","up"]},{"osd":2,"uuid":"b83486bb-27f2-4d56-97fb-fb336fec4b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6801","nonce":3597250789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6803","nonce":3597250789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6807","nonce":3597250789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3597250789},{"type":"v1","addr":"172.21.15.178:6805","nonce":3597250789}]},"public_addr":"172.21.15.178:6801/3597250789","cluster_addr":"172.21.15.178:6803/3597250789","heartbeat_back_addr":"172.21.15.178:6807/3597250789","heartbeat_front_addr":"172.21.15.178:6805/3597250789","state":["exists","up"]},{"osd":3,"uuid":"870e5bb6-eed5-4a0a-8e03-38de4f816b44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6811","nonce":3299043821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6813","nonce":3299043821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6817","nonce":3299043821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3299043821},{"type":"v1","addr":"172.21.15.90:6815","nonce":3299043821}]},"public_addr":"172.21.15.90:6811/3299043821","cluster_addr":"172.21.15.90:6813/3299043821","heartbeat_back_addr":"172.21.15.90:6817/3299043821","heartbeat_front_addr":"172.21.15.90:6815/3299043821","state":["exists","up"]},{"osd":4,"uuid":"cb9f7f48-76e0-4368-ab87-2a09a8487ccd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6809","nonce":3548915687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6811","nonce":3548915687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6815","nonce":3548915687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":3548915687},{"type":"v1","addr":"172.21.15.176:6813","nonce":3548915687}]},"public_addr":"172.21.15.176:6809/3548915687","cluster_addr":"172.21.15.176:6811/3548915687","heartbeat_back_addr":"172.21.15.176:6815/3548915687","heartbeat_front_addr":"172.21.15.176:6813/3548915687","state":["exists","up"]},{"osd":5,"uuid":"54749022-c50d-4831-a1d6-8412b59c8a01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6809","nonce":286274518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6811","nonce":286274518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6815","nonce":286274518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":286274518},{"type":"v1","addr":"172.21.15.178:6813","nonce":286274518}]},"public_addr":"172.21.15.178:6809/286274518","cluster_addr":"172.21.15.178:6811/286274518","heartbeat_back_addr":"172.21.15.178:6815/286274518","heartbeat_front_addr":"172.21.15.178:6813/286274518","state":["exists","up"]},{"osd":6,"uuid":"09bb5393-2efd-4a63-b0e8-99bef5bced13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6819","nonce":67640620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6821","nonce":67640620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6825","nonce":67640620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":67640620},{"type":"v1","addr":"172.21.15.90:6823","nonce":67640620}]},"public_addr":"172.21.15.90:6819/67640620","cluster_addr":"172.21.15.90:6821/67640620","heartbeat_back_addr":"172.21.15.90:6825/67640620","heartbeat_front_addr":"172.21.15.90:6823/67640620","state":["exists","up"]},{"osd":7,"uuid":"7a859e69-6b63-467b-a3b3-99247d66f5c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6817","nonce":3998561598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6819","nonce":3998561598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6823","nonce":3998561598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":3998561598},{"type":"v1","addr":"172.21.15.176:6821","nonce":3998561598}]},"public_addr":"172.21.15.176:6817/3998561598","cluster_addr":"172.21.15.176:6819/3998561598","heartbeat_back_addr":"172.21.15.176:6823/3998561598","heartbeat_front_addr":"172.21.15.176:6821/3998561598","state":["exists","up"]},{"osd":8,"uuid":"bc23144e-22a5-4894-a8b5-9323383574d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6817","nonce":490229963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6819","nonce":490229963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6823","nonce":490229963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":490229963},{"type":"v1","addr":"172.21.15.178:6821","nonce":490229963}]},"public_addr":"172.21.15.178:6817/490229963","cluster_addr":"172.21.15.178:6819/490229963","heartbeat_back_addr":"172.21.15.178:6823/490229963","heartbeat_front_addr":"172.21.15.178:6821/490229963","state":["exists","up"]},{"osd":9,"uuid":"0c2cecd4-55de-4fc4-a6f4-ca97c8460444","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6827","nonce":2061394761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6829","nonce":2061394761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6833","nonce":2061394761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":2061394761},{"type":"v1","addr":"172.21.15.90:6831","nonce":2061394761}]},"public_addr":"172.21.15.90:6827/2061394761","cluster_addr":"172.21.15.90:6829/2061394761","heartbeat_back_addr":"172.21.15.90:6833/2061394761","heartbeat_front_addr":"172.21.15.90:6831/2061394761","state":["exists","up"]},{"osd":10,"uuid":"fa04232b-b1ec-447a-8e94-7c9bf26d0291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6825","nonce":639411757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6827","nonce":639411757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6831","nonce":639411757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":639411757},{"type":"v1","addr":"172.21.15.176:6829","nonce":639411757}]},"public_addr":"172.21.15.176:6825/639411757","cluster_addr":"172.21.15.176:6827/639411757","heartbeat_back_addr":"172.21.15.176:6831/639411757","heartbeat_front_addr":"172.21.15.176:6829/639411757","state":["exists","up"]},{"osd":11,"uuid":"7282e294-64fc-4856-9c74-86115f69408c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6825","nonce":4144536256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6827","nonce":4144536256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6831","nonce":4144536256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":4144536256},{"type":"v1","addr":"172.21.15.178:6829","nonce":4144536256}]},"public_addr":"172.21.15.178:6825/4144536256","cluster_addr":"172.21.15.178:6827/4144536256","heartbeat_back_addr":"172.21.15.178:6831/4144536256","heartbeat_front_addr":"172.21.15.178:6829/4144536256","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:42:44.280345+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:01.235630+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:19.408896+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:36.201749+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:43:53.401854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:44:11.342158+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:44:29.324502+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:44:46.821595+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:04.642344+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:23.019004+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:41.140149+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:45:59.401951+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/159655312":"2024-02-22T23:41:33.360021+0000","172.21.15.90:6801/1385551604":"2024-02-22T23:41:33.360021+0000","172.21.15.90:0/1452574770":"2024-02-22T23:41:33.360021+0000","172.21.15.90:0/3815292343":"2024-02-22T23:41:11.484145+0000","172.21.15.90:6800/1385551604":"2024-02-22T23:41:33.360021+0000","172.21.15.90:6801/4247695574":"2024-02-22T23:41:11.484145+0000","172.21.15.90:0/1328266622":"2024-02-22T23:41:11.484145+0000","172.21.15.90:6800/4247695574":"2024-02-22T23:41:11.484145+0000","172.21.15.90:0/2569421522":"2024-02-22T23:41:11.484145+0000","172.21.15.90:0/3019701027":"2024-02-22T23:41:33.360021+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:46:56.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.409+0000 7fd301333700 1 -- 172.21.15.90:0/3316159251 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fd2e40a1120 msgr2=0x7fd2e40a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:56.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.409+0000 7fd301333700 1 --2- 172.21.15.90:0/3316159251 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fd2e40a1120 0x7fd2e40a35e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0005fd0 tx=0x7fd2f0005f00 comp rx=0 tx=0).stop 2024-02-21T23:46:56.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.409+0000 7fd301333700 1 -- 172.21.15.90:0/3316159251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2fc102910 msgr2=0x7fd2fc19dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:56.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.409+0000 7fd301333700 1 --2- 172.21.15.90:0/3316159251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2fc102910 0x7fd2fc19dfe0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd2f4007d70 tx=0x7fd2f400a340 comp rx=0 tx=0).stop 2024-02-21T23:46:56.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.409+0000 7fd301333700 1 -- 172.21.15.90:0/3316159251 shutdown_connections 2024-02-21T23:46:56.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.409+0000 7fd301333700 1 --2- 172.21.15.90:0/3316159251 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fd2e40a1120 0x7fd2e40a35e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:56.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.409+0000 7fd301333700 1 --2- 172.21.15.90:0/3316159251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2fc102910 0x7fd2fc19dfe0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:56.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.409+0000 7fd301333700 1 --2- 172.21.15.90:0/3316159251 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2fc0fef00 0x7fd2fc196a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:56.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.409+0000 7fd301333700 1 --2- 172.21.15.90:0/3316159251 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd2fc101f70 0x7fd2fc196fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:56.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.409+0000 7fd301333700 1 -- 172.21.15.90:0/3316159251 >> 172.21.15.90:0/3316159251 conn(0x7fd2fc0fa810 msgr2=0x7fd2fc105270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:56.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.409+0000 7fd301333700 1 -- 172.21.15.90:0/3316159251 shutdown_connections 2024-02-21T23:46:56.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.409+0000 7fd301333700 1 -- 172.21.15.90:0/3316159251 wait complete. 2024-02-21T23:46:56.407 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-02-21T23:46:56.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.548+0000 7f8cd12b9700 1 -- 172.21.15.90:0/297616208 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ccc101780 msgr2=0x7f8ccc108030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:56.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.548+0000 7f8cd12b9700 1 --2- 172.21.15.90:0/297616208 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ccc101780 0x7f8ccc108030 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8cc400b0d0 tx=0x7f8cc400b3e0 comp rx=0 tx=0).stop 2024-02-21T23:46:56.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.548+0000 7f8cd12b9700 1 -- 172.21.15.90:0/297616208 shutdown_connections 2024-02-21T23:46:56.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.548+0000 7f8cd12b9700 1 --2- 172.21.15.90:0/297616208 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ccc101780 0x7f8ccc108030 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:56.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.548+0000 7f8cd12b9700 1 --2- 172.21.15.90:0/297616208 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ccc0ffbf0 0x7f8ccc100010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:56.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.548+0000 7f8cd12b9700 1 --2- 172.21.15.90:0/297616208 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ccc100de0 0x7f8ccc101240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:56.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.548+0000 7f8cd12b9700 1 -- 172.21.15.90:0/297616208 >> 172.21.15.90:0/297616208 conn(0x7f8ccc0fb160 msgr2=0x7f8ccc0fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:56.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.548+0000 7f8cd12b9700 1 -- 172.21.15.90:0/297616208 shutdown_connections 2024-02-21T23:46:56.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.548+0000 7f8cd12b9700 1 -- 172.21.15.90:0/297616208 wait complete. 2024-02-21T23:46:56.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.549+0000 7f8cd12b9700 1 Processor -- start 2024-02-21T23:46:56.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.549+0000 7f8cd12b9700 1 -- start start 2024-02-21T23:46:56.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.549+0000 7f8cd12b9700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ccc0ffbf0 0x7f8ccc195c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:56.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.549+0000 7f8cd12b9700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ccc100de0 0x7f8ccc1961c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:56.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.549+0000 7f8cd12b9700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ccc101780 0x7f8ccc19d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:56.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.549+0000 7f8cd12b9700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ccc19d770 con 0x7f8ccc0ffbf0 2024-02-21T23:46:56.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.549+0000 7f8cd12b9700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ccc19d8e0 con 0x7f8ccc100de0 2024-02-21T23:46:56.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.549+0000 7f8cd12b9700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ccc19daa0 con 0x7f8ccc101780 2024-02-21T23:46:56.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.549+0000 7f8ccaffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ccc0ffbf0 0x7f8ccc195c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:56.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.549+0000 7f8ccaffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ccc0ffbf0 0x7f8ccc195c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59606/0 (socket says 172.21.15.90:59606) 2024-02-21T23:46:56.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.549+0000 7f8ccaffd700 1 -- 172.21.15.90:0/322853564 learned_addr learned my addr 172.21.15.90:0/322853564 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:56.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.550+0000 7f8cca7fc700 1 --2- 172.21.15.90:0/322853564 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ccc100de0 0x7f8ccc1961c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:56.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.550+0000 7f8ccb7fe700 1 --2- 172.21.15.90:0/322853564 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ccc101780 0x7f8ccc19d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:56.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.550+0000 7f8ccaffd700 1 -- 172.21.15.90:0/322853564 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ccc101780 msgr2=0x7f8ccc19d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:56.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.550+0000 7f8ccaffd700 1 --2- 172.21.15.90:0/322853564 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ccc101780 0x7f8ccc19d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:56.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.550+0000 7f8ccaffd700 1 -- 172.21.15.90:0/322853564 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ccc100de0 msgr2=0x7f8ccc1961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:56.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.550+0000 7f8ccaffd700 1 --2- 172.21.15.90:0/322853564 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ccc100de0 0x7f8ccc1961c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:56.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.550+0000 7f8ccaffd700 1 -- 172.21.15.90:0/322853564 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc4009d00 con 0x7f8ccc0ffbf0 2024-02-21T23:46:56.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.550+0000 7f8ccaffd700 1 --2- 172.21.15.90:0/322853564 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ccc0ffbf0 0x7f8ccc195c80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc00d980 tx=0x7f8cbc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:56.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.550+0000 7f8cb3fff700 1 -- 172.21.15.90:0/322853564 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc004020 con 0x7f8ccc0ffbf0 2024-02-21T23:46:56.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.550+0000 7f8cd12b9700 1 -- 172.21.15.90:0/322853564 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ccc19dd80 con 0x7f8ccc0ffbf0 2024-02-21T23:46:56.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.550+0000 7f8cb3fff700 1 -- 172.21.15.90:0/322853564 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8cbc009730 con 0x7f8ccc0ffbf0 2024-02-21T23:46:56.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.550+0000 7f8cb3fff700 1 -- 172.21.15.90:0/322853564 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc003a30 con 0x7f8ccc0ffbf0 2024-02-21T23:46:56.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.550+0000 7f8cd12b9700 1 -- 172.21.15.90:0/322853564 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ccc19e220 con 0x7f8ccc0ffbf0 2024-02-21T23:46:56.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.551+0000 7f8cd12b9700 1 -- 172.21.15.90:0/322853564 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ccc106fa0 con 0x7f8ccc0ffbf0 2024-02-21T23:46:56.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.552+0000 7f8cb3fff700 1 -- 172.21.15.90:0/322853564 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7f8cbc0098a0 con 0x7f8ccc0ffbf0 2024-02-21T23:46:56.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.553+0000 7f8cb3fff700 1 --2- 172.21.15.90:0/322853564 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f8cb40a1120 0x7f8cb40a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:56.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.553+0000 7f8cca7fc700 1 --2- 172.21.15.90:0/322853564 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f8cb40a1120 0x7f8cb40a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:56.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.553+0000 7f8cb3fff700 1 -- 172.21.15.90:0/322853564 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f8cbc0cc7e0 con 0x7f8ccc0ffbf0 2024-02-21T23:46:56.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.553+0000 7f8cca7fc700 1 --2- 172.21.15.90:0/322853564 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f8cb40a1120 0x7f8cb40a35e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8cc0009c60 tx=0x7f8cc0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:56.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.557+0000 7f8cb3fff700 1 -- 172.21.15.90:0/322853564 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f8cbc095d20 con 0x7f8ccc0ffbf0 2024-02-21T23:46:56.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:56 smithi090 ceph-mon[43385]: pgmap v236: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-02-21T23:46:56.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:56 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/4005308737' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-21T23:46:56.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:56 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/3316159251' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-21T23:46:56.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:56.800+0000 7f8cd12b9700 1 -- 172.21.15.90:0/322853564 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f8ccc02ce20 con 0x7f8ccc0ffbf0 2024-02-21T23:46:56.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:56 smithi176 ceph-mon[47172]: pgmap v236: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-02-21T23:46:56.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:56 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/4005308737' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-21T23:46:56.835 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:56 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/3316159251' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-21T23:46:56.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:56 smithi178 ceph-mon[47547]: pgmap v236: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-02-21T23:46:56.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:56 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/4005308737' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-21T23:46:56.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:56 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/3316159251' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-21T23:46:57.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.785+0000 7f8cb3fff700 1 -- 172.21.15.90:0/322853564 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) v1 ==== 118+0+0 (secure 0 0 0) 0x7f8cbc099340 con 0x7f8ccc0ffbf0 2024-02-21T23:46:57.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.788+0000 7f8cd12b9700 1 -- 172.21.15.90:0/322853564 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f8cb40a1120 msgr2=0x7f8cb40a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:57.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.788+0000 7f8cd12b9700 1 --2- 172.21.15.90:0/322853564 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f8cb40a1120 0x7f8cb40a35e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8cc0009c60 tx=0x7f8cc0009260 comp rx=0 tx=0).stop 2024-02-21T23:46:57.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.788+0000 7f8cd12b9700 1 -- 172.21.15.90:0/322853564 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ccc0ffbf0 msgr2=0x7f8ccc195c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:57.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.788+0000 7f8cd12b9700 1 --2- 172.21.15.90:0/322853564 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ccc0ffbf0 0x7f8ccc195c80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc00d980 tx=0x7f8cbc00dc90 comp rx=0 tx=0).stop 2024-02-21T23:46:57.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.788+0000 7f8cd12b9700 1 -- 172.21.15.90:0/322853564 shutdown_connections 2024-02-21T23:46:57.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.788+0000 7f8cd12b9700 1 --2- 172.21.15.90:0/322853564 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f8cb40a1120 0x7f8cb40a35e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:57.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.788+0000 7f8cd12b9700 1 --2- 172.21.15.90:0/322853564 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ccc101780 0x7f8ccc19d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:57.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.788+0000 7f8cd12b9700 1 --2- 172.21.15.90:0/322853564 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ccc0ffbf0 0x7f8ccc195c80 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:57.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.788+0000 7f8cd12b9700 1 --2- 172.21.15.90:0/322853564 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ccc100de0 0x7f8ccc1961c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:57.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.788+0000 7f8cd12b9700 1 -- 172.21.15.90:0/322853564 >> 172.21.15.90:0/322853564 conn(0x7f8ccc0fb160 msgr2=0x7f8ccc1040e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:57.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.788+0000 7f8cd12b9700 1 -- 172.21.15.90:0/322853564 shutdown_connections 2024-02-21T23:46:57.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.788+0000 7f8cd12b9700 1 -- 172.21.15.90:0/322853564 wait complete. 2024-02-21T23:46:57.788 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 1, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-02-21T23:46:57.789 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-02-21T23:46:57.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.936+0000 7f636080e700 1 -- 172.21.15.90:0/962936002 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f635c0fff40 msgr2=0x7f635c1003c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:57.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.936+0000 7f636080e700 1 --2- 172.21.15.90:0/962936002 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f635c0fff40 0x7f635c1003c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f634c0099c0 tx=0x7f634c009cd0 comp rx=0 tx=0).stop 2024-02-21T23:46:57.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.936+0000 7f636080e700 1 -- 172.21.15.90:0/962936002 shutdown_connections 2024-02-21T23:46:57.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.936+0000 7f636080e700 1 --2- 172.21.15.90:0/962936002 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f635c100900 0x7f635c100d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:57.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.936+0000 7f636080e700 1 --2- 172.21.15.90:0/962936002 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f635c0ff5e0 0x7f635c0ffa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:57.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.936+0000 7f636080e700 1 --2- 172.21.15.90:0/962936002 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f635c0fff40 0x7f635c1003c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:57.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.936+0000 7f636080e700 1 -- 172.21.15.90:0/962936002 >> 172.21.15.90:0/962936002 conn(0x7f635c0fb180 msgr2=0x7f635c0fd5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:57.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.936+0000 7f636080e700 1 -- 172.21.15.90:0/962936002 shutdown_connections 2024-02-21T23:46:57.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.937+0000 7f636080e700 1 -- 172.21.15.90:0/962936002 wait complete. 2024-02-21T23:46:57.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.937+0000 7f636080e700 1 Processor -- start 2024-02-21T23:46:57.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.937+0000 7f636080e700 1 -- start start 2024-02-21T23:46:57.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.937+0000 7f636080e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f635c0ff5e0 0x7f635c198e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:57.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.937+0000 7f636080e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f635c0fff40 0x7f635c199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:57.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.937+0000 7f636080e700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f635c100900 0x7f635c1a0390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:57.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.937+0000 7f636080e700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635c1a0960 con 0x7f635c0fff40 2024-02-21T23:46:57.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.937+0000 7f636080e700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635c1a0aa0 con 0x7f635c0ff5e0 2024-02-21T23:46:57.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.937+0000 7f636080e700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635c1a0c30 con 0x7f635c100900 2024-02-21T23:46:57.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.937+0000 7f6359d9b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f635c0fff40 0x7f635c199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:57.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.938+0000 7f6359d9b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f635c0fff40 0x7f635c199370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59628/0 (socket says 172.21.15.90:59628) 2024-02-21T23:46:57.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.938+0000 7f6359d9b700 1 -- 172.21.15.90:0/392464424 learned_addr learned my addr 172.21.15.90:0/392464424 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:57.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.938+0000 7f635a59c700 1 --2- 172.21.15.90:0/392464424 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f635c0ff5e0 0x7f635c198e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:57.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.938+0000 7f6359d9b700 1 -- 172.21.15.90:0/392464424 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f635c100900 msgr2=0x7f635c1a0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:57.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.938+0000 7f6359d9b700 1 --2- 172.21.15.90:0/392464424 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f635c100900 0x7f635c1a0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:57.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.938+0000 7f6359d9b700 1 -- 172.21.15.90:0/392464424 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f635c0ff5e0 msgr2=0x7f635c198e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:57.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.938+0000 7f6359d9b700 1 --2- 172.21.15.90:0/392464424 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f635c0ff5e0 0x7f635c198e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:57.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.938+0000 7f6359d9b700 1 -- 172.21.15.90:0/392464424 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f634c0095f0 con 0x7f635c0fff40 2024-02-21T23:46:57.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.938+0000 7f635a59c700 1 --2- 172.21.15.90:0/392464424 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f635c0ff5e0 0x7f635c198e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:46:57.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.938+0000 7f6359d9b700 1 --2- 172.21.15.90:0/392464424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f635c0fff40 0x7f635c199370 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f634c0036f0 tx=0x7f634c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:57.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.938+0000 7f634b7fe700 1 -- 172.21.15.90:0/392464424 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634c01d070 con 0x7f635c0fff40 2024-02-21T23:46:57.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.938+0000 7f634b7fe700 1 -- 172.21.15.90:0/392464424 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f634c004110 con 0x7f635c0fff40 2024-02-21T23:46:57.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.938+0000 7f636080e700 1 -- 172.21.15.90:0/392464424 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f635c1a0eb0 con 0x7f635c0fff40 2024-02-21T23:46:57.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.938+0000 7f634b7fe700 1 -- 172.21.15.90:0/392464424 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634c022e10 con 0x7f635c0fff40 2024-02-21T23:46:57.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.938+0000 7f636080e700 1 -- 172.21.15.90:0/392464424 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f635c1a12d0 con 0x7f635c0fff40 2024-02-21T23:46:57.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.939+0000 7f636080e700 1 -- 172.21.15.90:0/392464424 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f635c10a3c0 con 0x7f635c0fff40 2024-02-21T23:46:57.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.940+0000 7f634b7fe700 1 -- 172.21.15.90:0/392464424 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7f634c022470 con 0x7f635c0fff40 2024-02-21T23:46:57.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.941+0000 7f634b7fe700 1 --2- 172.21.15.90:0/392464424 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f63440a1120 0x7f63440a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:57.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.941+0000 7f635a59c700 1 --2- 172.21.15.90:0/392464424 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f63440a1120 0x7f63440a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:57.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.941+0000 7f634b7fe700 1 -- 172.21.15.90:0/392464424 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f634c0cde80 con 0x7f635c0fff40 2024-02-21T23:46:57.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.941+0000 7f635a59c700 1 --2- 172.21.15.90:0/392464424 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f63440a1120 0x7f63440a35e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6350005950 tx=0x7f63500058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:57.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:57.945+0000 7f634b7fe700 1 -- 172.21.15.90:0/392464424 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f634c05d960 con 0x7f635c0fff40 2024-02-21T23:46:58.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:57 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/322853564' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-02-21T23:46:58.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:57 smithi176 ceph-mon[47172]: pgmap v237: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-02-21T23:46:58.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:57 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/322853564' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-02-21T23:46:58.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:57 smithi178 ceph-mon[47547]: pgmap v237: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-02-21T23:46:58.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.190+0000 7f636080e700 1 -- 172.21.15.90:0/392464424 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f635c02cea0 con 0x7f635c0fff40 2024-02-21T23:46:58.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:57 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/322853564' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-02-21T23:46:58.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:57 smithi090 ceph-mon[43385]: pgmap v237: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-02-21T23:46:58.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.787+0000 7f634b7fe700 1 -- 172.21.15.90:0/392464424 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v19) v1 ==== 121+0+0 (secure 0 0 0) 0x7f634c09fe60 con 0x7f635c0fff40 2024-02-21T23:46:58.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.789+0000 7f636080e700 1 -- 172.21.15.90:0/392464424 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f63440a1120 msgr2=0x7f63440a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:58.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.789+0000 7f636080e700 1 --2- 172.21.15.90:0/392464424 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f63440a1120 0x7f63440a35e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6350005950 tx=0x7f63500058e0 comp rx=0 tx=0).stop 2024-02-21T23:46:58.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.789+0000 7f636080e700 1 -- 172.21.15.90:0/392464424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f635c0fff40 msgr2=0x7f635c199370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:58.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.789+0000 7f636080e700 1 --2- 172.21.15.90:0/392464424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f635c0fff40 0x7f635c199370 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f634c0036f0 tx=0x7f634c0047b0 comp rx=0 tx=0).stop 2024-02-21T23:46:58.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.789+0000 7f636080e700 1 -- 172.21.15.90:0/392464424 shutdown_connections 2024-02-21T23:46:58.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.789+0000 7f636080e700 1 --2- 172.21.15.90:0/392464424 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7f63440a1120 0x7f63440a35e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:58.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.789+0000 7f636080e700 1 --2- 172.21.15.90:0/392464424 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f635c100900 0x7f635c1a0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:58.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.789+0000 7f636080e700 1 --2- 172.21.15.90:0/392464424 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f635c0ff5e0 0x7f635c198e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:58.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.789+0000 7f636080e700 1 --2- 172.21.15.90:0/392464424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f635c0fff40 0x7f635c199370 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:58.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.789+0000 7f636080e700 1 -- 172.21.15.90:0/392464424 >> 172.21.15.90:0/392464424 conn(0x7f635c0fb180 msgr2=0x7f635c0fcc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:58.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.789+0000 7f636080e700 1 -- 172.21.15.90:0/392464424 shutdown_connections 2024-02-21T23:46:58.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.789+0000 7f636080e700 1 -- 172.21.15.90:0/392464424 wait complete. 2024-02-21T23:46:58.788 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-02-21T23:46:58.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.929+0000 7fcfae05c700 1 -- 172.21.15.90:0/810621036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfa8100d10 msgr2=0x7fcfa8101130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:58.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.929+0000 7fcfae05c700 1 --2- 172.21.15.90:0/810621036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfa8100d10 0x7fcfa8101130 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fcf980099c0 tx=0x7fcf98009cd0 comp rx=0 tx=0).stop 2024-02-21T23:46:58.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.929+0000 7fcfae05c700 1 -- 172.21.15.90:0/810621036 shutdown_connections 2024-02-21T23:46:58.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.929+0000 7fcfae05c700 1 --2- 172.21.15.90:0/810621036 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfa81028a0 0x7fcfa8109150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:58.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.929+0000 7fcfae05c700 1 --2- 172.21.15.90:0/810621036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfa8100d10 0x7fcfa8101130 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:58.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.929+0000 7fcfae05c700 1 --2- 172.21.15.90:0/810621036 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcfa8101f00 0x7fcfa8102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:58.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.929+0000 7fcfae05c700 1 -- 172.21.15.90:0/810621036 >> 172.21.15.90:0/810621036 conn(0x7fcfa80fc280 msgr2=0x7fcfa80fe6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:58.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.929+0000 7fcfae05c700 1 -- 172.21.15.90:0/810621036 shutdown_connections 2024-02-21T23:46:58.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.929+0000 7fcfae05c700 1 -- 172.21.15.90:0/810621036 wait complete. 2024-02-21T23:46:58.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.929+0000 7fcfae05c700 1 Processor -- start 2024-02-21T23:46:58.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.929+0000 7fcfae05c700 1 -- start start 2024-02-21T23:46:58.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.929+0000 7fcfae05c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfa8100d10 0x7fcfa8196cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:58.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.930+0000 7fcfae05c700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcfa8101f00 0x7fcfa8197200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:58.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.930+0000 7fcfae05c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfa81028a0 0x7fcfa819e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:58.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.930+0000 7fcfae05c700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa819e7b0 con 0x7fcfa81028a0 2024-02-21T23:46:58.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.930+0000 7fcfae05c700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa819e920 con 0x7fcfa8100d10 2024-02-21T23:46:58.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.930+0000 7fcfae05c700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa819eae0 con 0x7fcfa8101f00 2024-02-21T23:46:58.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.930+0000 7fcfa7fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfa81028a0 0x7fcfa819e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:58.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.930+0000 7fcfa7fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfa81028a0 0x7fcfa819e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43098/0 (socket says 172.21.15.90:43098) 2024-02-21T23:46:58.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.930+0000 7fcfa7fff700 1 -- 172.21.15.90:0/1873413406 learned_addr learned my addr 172.21.15.90:0/1873413406 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-21T23:46:58.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.930+0000 7fcfa6ffd700 1 --2- 172.21.15.90:0/1873413406 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcfa8101f00 0x7fcfa8197200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:58.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.930+0000 7fcfa77fe700 1 --2- 172.21.15.90:0/1873413406 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfa8100d10 0x7fcfa8196cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:58.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.930+0000 7fcfa7fff700 1 -- 172.21.15.90:0/1873413406 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcfa8101f00 msgr2=0x7fcfa8197200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:58.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.930+0000 7fcfa7fff700 1 --2- 172.21.15.90:0/1873413406 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcfa8101f00 0x7fcfa8197200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:58.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.930+0000 7fcfa7fff700 1 -- 172.21.15.90:0/1873413406 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfa8100d10 msgr2=0x7fcfa8196cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:58.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.930+0000 7fcfa7fff700 1 --2- 172.21.15.90:0/1873413406 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfa8100d10 0x7fcfa8196cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:58.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.930+0000 7fcfa7fff700 1 -- 172.21.15.90:0/1873413406 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf980095f0 con 0x7fcfa81028a0 2024-02-21T23:46:58.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.930+0000 7fcfa77fe700 1 --2- 172.21.15.90:0/1873413406 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfa8100d10 0x7fcfa8196cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:46:58.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.930+0000 7fcfa7fff700 1 --2- 172.21.15.90:0/1873413406 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfa81028a0 0x7fcfa819e1e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0007ee0 tx=0x7fcfa000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:58.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.930+0000 7fcfa4ff9700 1 -- 172.21.15.90:0/1873413406 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa000ddd0 con 0x7fcfa81028a0 2024-02-21T23:46:58.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.930+0000 7fcfa4ff9700 1 -- 172.21.15.90:0/1873413406 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcfa000f040 con 0x7fcfa81028a0 2024-02-21T23:46:58.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.930+0000 7fcfae05c700 1 -- 172.21.15.90:0/1873413406 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfa819edc0 con 0x7fcfa81028a0 2024-02-21T23:46:58.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.930+0000 7fcfa4ff9700 1 -- 172.21.15.90:0/1873413406 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa00048e0 con 0x7fcfa81028a0 2024-02-21T23:46:58.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.930+0000 7fcfae05c700 1 -- 172.21.15.90:0/1873413406 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfa819f1c0 con 0x7fcfa81028a0 2024-02-21T23:46:58.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.931+0000 7fcfae05c700 1 -- 172.21.15.90:0/1873413406 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfa81081e0 con 0x7fcfa81028a0 2024-02-21T23:46:58.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.932+0000 7fcfa4ff9700 1 -- 172.21.15.90:0/1873413406 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 135877+0+0 (secure 0 0 0) 0x7fcfa0003a00 con 0x7fcfa81028a0 2024-02-21T23:46:58.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.933+0000 7fcfa4ff9700 1 --2- 172.21.15.90:0/1873413406 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fcf900a1120 0x7fcf900a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:46:58.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.933+0000 7fcfa77fe700 1 --2- 172.21.15.90:0/1873413406 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fcf900a1120 0x7fcf900a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:46:58.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.933+0000 7fcfa4ff9700 1 -- 172.21.15.90:0/1873413406 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fcfa00ccc00 con 0x7fcfa81028a0 2024-02-21T23:46:58.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.933+0000 7fcfa77fe700 1 --2- 172.21.15.90:0/1873413406 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fcf900a1120 0x7fcf900a35e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8198140 tx=0x7fcf9801a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:46:58.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:58.937+0000 7fcfa4ff9700 1 -- 172.21.15.90:0/1873413406 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fcfa00961a0 con 0x7fcfa81028a0 2024-02-21T23:46:59.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:58 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/322853564' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-02-21T23:46:59.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:58 smithi176 ceph-mon[47172]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-02-21T23:46:59.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:58 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/392464424' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-02-21T23:46:59.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:59.184+0000 7fcfae05c700 1 -- 172.21.15.90:0/1873413406 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7fcfa8067a80 con 0x7fcfa81028a0 2024-02-21T23:46:59.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:58 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/322853564' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-02-21T23:46:59.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:58 smithi178 ceph-mon[47547]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-02-21T23:46:59.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:58 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/392464424' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-02-21T23:46:59.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:58 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/322853564' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-02-21T23:46:59.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:58 smithi090 ceph-mon[43385]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-02-21T23:46:59.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:58 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/392464424' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-02-21T23:46:59.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:59.796+0000 7fcfa4ff9700 1 -- 172.21.15.90:0/1873413406 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v21) v1 ==== 115+0+0 (secure 0 0 0) 0x7fcfa00230b0 con 0x7fcfa81028a0 2024-02-21T23:46:59.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:59.799+0000 7fcfae05c700 1 -- 172.21.15.90:0/1873413406 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fcf900a1120 msgr2=0x7fcf900a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:59.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:59.799+0000 7fcfae05c700 1 --2- 172.21.15.90:0/1873413406 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fcf900a1120 0x7fcf900a35e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8198140 tx=0x7fcf9801a040 comp rx=0 tx=0).stop 2024-02-21T23:46:59.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:59.799+0000 7fcfae05c700 1 -- 172.21.15.90:0/1873413406 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfa81028a0 msgr2=0x7fcfa819e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:46:59.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:59.799+0000 7fcfae05c700 1 --2- 172.21.15.90:0/1873413406 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfa81028a0 0x7fcfa819e1e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0007ee0 tx=0x7fcfa000d3b0 comp rx=0 tx=0).stop 2024-02-21T23:46:59.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:59.799+0000 7fcfae05c700 1 -- 172.21.15.90:0/1873413406 shutdown_connections 2024-02-21T23:46:59.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:59.799+0000 7fcfae05c700 1 --2- 172.21.15.90:0/1873413406 >> [v2:172.21.15.90:6800/3697394685,v1:172.21.15.90:6801/3697394685] conn(0x7fcf900a1120 0x7fcf900a35e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:59.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:59.799+0000 7fcfae05c700 1 --2- 172.21.15.90:0/1873413406 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfa81028a0 0x7fcfa819e1e0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:59.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:59.799+0000 7fcfae05c700 1 --2- 172.21.15.90:0/1873413406 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfa8100d10 0x7fcfa8196cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:59.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:59.799+0000 7fcfae05c700 1 --2- 172.21.15.90:0/1873413406 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcfa8101f00 0x7fcfa8197200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:46:59.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:59.799+0000 7fcfae05c700 1 -- 172.21.15.90:0/1873413406 >> 172.21.15.90:0/1873413406 conn(0x7fcfa80fc280 msgr2=0x7fcfa8105320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:46:59.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:59.799+0000 7fcfae05c700 1 -- 172.21.15.90:0/1873413406 shutdown_connections 2024-02-21T23:46:59.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-21T23:46:59.799+0000 7fcfae05c700 1 -- 172.21.15.90:0/1873413406 wait complete. 2024-02-21T23:46:59.797 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-02-21T23:46:59.798 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c1dc18f444603397768fcb7c34a8488b10284df6/qa/tasks/cephadm.py", line 1051, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/github.com_ceph_ceph-c_c1dc18f444603397768fcb7c34a8488b10284df6/qa/tasks/cephfs/filesystem.py", line 532, in __init__ self.create() File "/home/teuthworker/src/github.com_ceph_ceph-c_c1dc18f444603397768fcb7c34a8488b10284df6/qa/tasks/cephfs/filesystem.py", line 762, in create self.run_ceph_cmd('fs', 'subvolumegroup', 'create', self.name, 'qa') AttributeError: 'Filesystem' object has no attribute 'run_ceph_cmd' 2024-02-21T23:46:59.800 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-21T23:46:59.828 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-21T23:46:59.867 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-21T23:46:59.906 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-21T23:46:59.906 DEBUG:teuthology.orchestra.run.smithi090:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-21T23:46:59.924 DEBUG:teuthology.orchestra.run.smithi176:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-21T23:46:59.943 DEBUG:teuthology.orchestra.run.smithi178:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-21T23:46:59.965 INFO:tasks.cephadm:Stopping all daemons... 2024-02-21T23:46:59.965 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-02-21T23:46:59.965 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-7ed81218-d112-11ee-95bd-87774f69a715@mon.a 2024-02-21T23:47:00.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:59 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/392464424' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-02-21T23:47:00.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:59 smithi176 ceph-mon[47172]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-02-21T23:47:00.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:59 smithi176 ceph-mon[47172]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-02-21T23:47:00.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:59 smithi176 ceph-mon[47172]: from='client.? 172.21.15.90:0/1873413406' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-02-21T23:47:00.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:46:59 smithi176 ceph-mon[47172]: pgmap v238: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-02-21T23:47:00.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:59 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/392464424' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-02-21T23:47:00.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:59 smithi178 ceph-mon[47547]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-02-21T23:47:00.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:59 smithi178 ceph-mon[47547]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-02-21T23:47:00.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:59 smithi178 ceph-mon[47547]: from='client.? 172.21.15.90:0/1873413406' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-02-21T23:47:00.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:46:59 smithi178 ceph-mon[47547]: pgmap v238: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-02-21T23:47:00.201 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:59 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/392464424' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-02-21T23:47:00.201 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:59 smithi090 ceph-mon[43385]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-02-21T23:47:00.201 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:59 smithi090 ceph-mon[43385]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-02-21T23:47:00.201 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:59 smithi090 ceph-mon[43385]: from='client.? 172.21.15.90:0/1873413406' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-02-21T23:47:00.201 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:46:59 smithi090 ceph-mon[43385]: pgmap v238: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-02-21T23:47:00.201 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:47:00 smithi090 systemd[1]: Stopping Ceph mon.a for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:47:00.201 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:47:00 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mon-a[43362]: 2024-02-21T23:47:00.214+0000 7faa4ab9c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-21T23:47:00.201 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:47:00 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mon-a[43362]: 2024-02-21T23:47:00.214+0000 7faa4ab9c700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-02-21T23:47:00.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Feb 21 23:47:00 smithi090 bash[98720]: ceph-7ed81218-d112-11ee-95bd-87774f69a715-mon-a 2024-02-21T23:47:00.787 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@mon.a.service' 2024-02-21T23:47:00.850 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-21T23:47:00.850 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-02-21T23:47:00.850 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-02-21T23:47:00.850 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-7ed81218-d112-11ee-95bd-87774f69a715@mon.b 2024-02-21T23:47:01.267 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:47:00 smithi176 systemd[1]: Stopping Ceph mon.b for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:47:01.585 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:47:01 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mon-b[47149]: 2024-02-21T23:47:01.282+0000 7f0dce763700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-21T23:47:01.585 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:47:01 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mon-b[47149]: 2024-02-21T23:47:01.282+0000 7f0dce763700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-02-21T23:47:01.585 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 21 23:47:01 smithi176 bash[84904]: ceph-7ed81218-d112-11ee-95bd-87774f69a715-mon-b 2024-02-21T23:47:01.792 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@mon.b.service' 2024-02-21T23:47:01.839 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-21T23:47:01.839 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-02-21T23:47:01.839 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-02-21T23:47:01.840 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-7ed81218-d112-11ee-95bd-87774f69a715@mon.c 2024-02-21T23:47:02.176 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:47:01 smithi178 systemd[1]: Stopping Ceph mon.c for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:47:02.176 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:47:02 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mon-c[47504]: 2024-02-21T23:47:02.097+0000 7f2afce53700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-21T23:47:02.176 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:47:02 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mon-c[47504]: 2024-02-21T23:47:02.097+0000 7f2afce53700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-02-21T23:47:02.675 INFO:journalctl@ceph.mon.c.smithi178.stdout:Feb 21 23:47:02 smithi178 bash[87081]: ceph-7ed81218-d112-11ee-95bd-87774f69a715-mon-c 2024-02-21T23:47:02.781 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@mon.c.service' 2024-02-21T23:47:02.863 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-21T23:47:02.863 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-02-21T23:47:02.863 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-02-21T23:47:02.863 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-7ed81218-d112-11ee-95bd-87774f69a715@mgr.x 2024-02-21T23:47:03.150 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:47:02 smithi090 systemd[1]: Stopping Ceph mgr.x for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:47:03.478 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:47:03 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x[43633]: 2024-02-21T23:47:03.164+0000 7fd1a65ed700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-21T23:47:03.479 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 21 23:47:03 smithi090 bash[99214]: ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-x 2024-02-21T23:47:03.780 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@mgr.x.service' 2024-02-21T23:47:03.821 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-21T23:47:03.821 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-02-21T23:47:03.821 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-02-21T23:47:03.821 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-7ed81218-d112-11ee-95bd-87774f69a715@mgr.y 2024-02-21T23:47:04.246 INFO:journalctl@ceph.mgr.y.smithi176.stdout:Feb 21 23:47:03 smithi176 systemd[1]: Stopping Ceph mgr.y for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:47:04.585 INFO:journalctl@ceph.mgr.y.smithi176.stdout:Feb 21 23:47:04 smithi176 bash[85138]: ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-y 2024-02-21T23:47:04.585 INFO:journalctl@ceph.mgr.y.smithi176.stdout:Feb 21 23:47:04 smithi176 systemd[1]: ceph-7ed81218-d112-11ee-95bd-87774f69a715@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-02-21T23:47:04.611 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@mgr.y.service' 2024-02-21T23:47:04.648 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-21T23:47:04.648 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-02-21T23:47:04.649 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-02-21T23:47:04.649 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-7ed81218-d112-11ee-95bd-87774f69a715@mgr.z 2024-02-21T23:47:05.130 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Feb 21 23:47:04 smithi178 systemd[1]: Stopping Ceph mgr.z for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:47:05.425 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Feb 21 23:47:05 smithi178 bash[87233]: ceph-7ed81218-d112-11ee-95bd-87774f69a715-mgr-z 2024-02-21T23:47:05.425 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Feb 21 23:47:05 smithi178 systemd[1]: ceph-7ed81218-d112-11ee-95bd-87774f69a715@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-02-21T23:47:05.586 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@mgr.z.service' 2024-02-21T23:47:05.624 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-21T23:47:05.625 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-02-21T23:47:05.625 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-02-21T23:47:05.625 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-7ed81218-d112-11ee-95bd-87774f69a715@osd.0 2024-02-21T23:47:05.978 INFO:journalctl@ceph.osd.0.smithi090.stdout:Feb 21 23:47:05 smithi090 systemd[1]: Stopping Ceph osd.0 for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:47:05.978 INFO:journalctl@ceph.osd.0.smithi090.stdout:Feb 21 23:47:05 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-0[61828]: 2024-02-21T23:47:05.888+0000 7ff86b6a9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-21T23:47:05.978 INFO:journalctl@ceph.osd.0.smithi090.stdout:Feb 21 23:47:05 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-0[61828]: 2024-02-21T23:47:05.888+0000 7ff86b6a9700 -1 osd.0 85 *** Got signal Terminated *** 2024-02-21T23:47:05.979 INFO:journalctl@ceph.osd.0.smithi090.stdout:Feb 21 23:47:05 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-0[61828]: 2024-02-21T23:47:05.888+0000 7ff86b6a9700 -1 osd.0 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-21T23:47:11.729 INFO:journalctl@ceph.osd.0.smithi090.stdout:Feb 21 23:47:11 smithi090 bash[99354]: ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-0 2024-02-21T23:47:12.927 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@osd.0.service' 2024-02-21T23:47:12.969 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-21T23:47:12.969 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-02-21T23:47:12.969 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-02-21T23:47:12.969 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-7ed81218-d112-11ee-95bd-87774f69a715@osd.1 2024-02-21T23:47:13.335 INFO:journalctl@ceph.osd.1.smithi176.stdout:Feb 21 23:47:13 smithi176 systemd[1]: Stopping Ceph osd.1 for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:47:13.336 INFO:journalctl@ceph.osd.1.smithi176.stdout:Feb 21 23:47:13 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-1[52138]: 2024-02-21T23:47:13.213+0000 7f7a67696700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-21T23:47:13.336 INFO:journalctl@ceph.osd.1.smithi176.stdout:Feb 21 23:47:13 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-1[52138]: 2024-02-21T23:47:13.213+0000 7f7a67696700 -1 osd.1 85 *** Got signal Terminated *** 2024-02-21T23:47:13.336 INFO:journalctl@ceph.osd.1.smithi176.stdout:Feb 21 23:47:13 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-1[52138]: 2024-02-21T23:47:13.213+0000 7f7a67696700 -1 osd.1 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-21T23:47:19.085 INFO:journalctl@ceph.osd.1.smithi176.stdout:Feb 21 23:47:18 smithi176 bash[85279]: ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-1 2024-02-21T23:47:20.078 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@osd.1.service' 2024-02-21T23:47:20.121 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-21T23:47:20.121 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-02-21T23:47:20.121 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-02-21T23:47:20.122 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-7ed81218-d112-11ee-95bd-87774f69a715@osd.2 2024-02-21T23:47:20.426 INFO:journalctl@ceph.osd.2.smithi178.stdout:Feb 21 23:47:20 smithi178 systemd[1]: Stopping Ceph osd.2 for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:47:20.426 INFO:journalctl@ceph.osd.2.smithi178.stdout:Feb 21 23:47:20 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-2[53361]: 2024-02-21T23:47:20.394+0000 7facd5947700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-21T23:47:20.426 INFO:journalctl@ceph.osd.2.smithi178.stdout:Feb 21 23:47:20 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-2[53361]: 2024-02-21T23:47:20.394+0000 7facd5947700 -1 osd.2 85 *** Got signal Terminated *** 2024-02-21T23:47:20.426 INFO:journalctl@ceph.osd.2.smithi178.stdout:Feb 21 23:47:20 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-2[53361]: 2024-02-21T23:47:20.394+0000 7facd5947700 -1 osd.2 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-21T23:47:26.426 INFO:journalctl@ceph.osd.2.smithi178.stdout:Feb 21 23:47:26 smithi178 bash[87376]: ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-2 2024-02-21T23:47:27.526 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@osd.2.service' 2024-02-21T23:47:27.567 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-21T23:47:27.568 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-02-21T23:47:27.568 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-02-21T23:47:27.568 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-7ed81218-d112-11ee-95bd-87774f69a715@osd.3 2024-02-21T23:47:27.979 INFO:journalctl@ceph.osd.3.smithi090.stdout:Feb 21 23:47:27 smithi090 systemd[1]: Stopping Ceph osd.3 for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:47:27.979 INFO:journalctl@ceph.osd.3.smithi090.stdout:Feb 21 23:47:27 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-3[70542]: 2024-02-21T23:47:27.828+0000 7f256da8f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-21T23:47:27.979 INFO:journalctl@ceph.osd.3.smithi090.stdout:Feb 21 23:47:27 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-3[70542]: 2024-02-21T23:47:27.828+0000 7f256da8f700 -1 osd.3 85 *** Got signal Terminated *** 2024-02-21T23:47:27.979 INFO:journalctl@ceph.osd.3.smithi090.stdout:Feb 21 23:47:27 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-3[70542]: 2024-02-21T23:47:27.828+0000 7f256da8f700 -1 osd.3 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-21T23:47:33.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:32 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:32.954+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:47:33.480 INFO:journalctl@ceph.osd.3.smithi090.stdout:Feb 21 23:47:33 smithi090 bash[99624]: ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-3 2024-02-21T23:47:33.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:33 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:33.142+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:47:33.676 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:33 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:33.266+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:05.990749+0000 front 2024-02-21T23:47:05.990685+0000 (oldest deadline 2024-02-21T23:47:32.991590+0000) 2024-02-21T23:47:34.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:33 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:33.977+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:47:34.431 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:34 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:34.293+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:08.122686+0000 front 2024-02-21T23:47:08.122651+0000 (oldest deadline 2024-02-21T23:47:34.022914+0000) 2024-02-21T23:47:34.432 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:34 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:34.115+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:47:34.676 INFO:journalctl@ceph.osd.5.smithi178.stdout:Feb 21 23:47:34 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-5[61968]: 2024-02-21T23:47:34.385+0000 7f9acb6f8700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.048910+0000 front 2024-02-21T23:47:07.048893+0000 (oldest deadline 2024-02-21T23:47:34.049556+0000) 2024-02-21T23:47:34.676 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:34 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:34.282+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:05.990749+0000 front 2024-02-21T23:47:05.990685+0000 (oldest deadline 2024-02-21T23:47:32.991590+0000) 2024-02-21T23:47:34.885 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@osd.3.service' 2024-02-21T23:47:34.926 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-21T23:47:34.926 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-02-21T23:47:34.926 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-02-21T23:47:34.927 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-7ed81218-d112-11ee-95bd-87774f69a715@osd.4 2024-02-21T23:47:35.176 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:34 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:34.829+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:47:35.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:34 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:34.957+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:47:35.336 INFO:journalctl@ceph.osd.4.smithi176.stdout:Feb 21 23:47:34 smithi176 systemd[1]: Stopping Ceph osd.4 for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:47:35.336 INFO:journalctl@ceph.osd.4.smithi176.stdout:Feb 21 23:47:35 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-4[60843]: 2024-02-21T23:47:35.200+0000 7f6128c7f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-21T23:47:35.336 INFO:journalctl@ceph.osd.4.smithi176.stdout:Feb 21 23:47:35 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-4[60843]: 2024-02-21T23:47:35.200+0000 7f6128c7f700 -1 osd.4 85 *** Got signal Terminated *** 2024-02-21T23:47:35.336 INFO:journalctl@ceph.osd.4.smithi176.stdout:Feb 21 23:47:35 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-4[60843]: 2024-02-21T23:47:35.200+0000 7f6128c7f700 -1 osd.4 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-21T23:47:35.479 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:35 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:35.162+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:47:35.479 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:35 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:35.254+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:08.122686+0000 front 2024-02-21T23:47:08.122651+0000 (oldest deadline 2024-02-21T23:47:34.022914+0000) 2024-02-21T23:47:35.676 INFO:journalctl@ceph.osd.5.smithi178.stdout:Feb 21 23:47:35 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-5[61968]: 2024-02-21T23:47:35.408+0000 7f9acb6f8700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.048910+0000 front 2024-02-21T23:47:07.048893+0000 (oldest deadline 2024-02-21T23:47:34.049556+0000) 2024-02-21T23:47:35.676 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:35 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:35.254+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:05.990749+0000 front 2024-02-21T23:47:05.990685+0000 (oldest deadline 2024-02-21T23:47:32.991590+0000) 2024-02-21T23:47:36.176 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:35 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:35.807+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:47:36.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:35 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:35.997+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:47:36.479 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:36 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:36.192+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:47:36.480 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:36 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:36.252+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:08.122686+0000 front 2024-02-21T23:47:08.122651+0000 (oldest deadline 2024-02-21T23:47:34.022914+0000) 2024-02-21T23:47:36.676 INFO:journalctl@ceph.osd.5.smithi178.stdout:Feb 21 23:47:36 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-5[61968]: 2024-02-21T23:47:36.445+0000 7f9acb6f8700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.048910+0000 front 2024-02-21T23:47:07.048893+0000 (oldest deadline 2024-02-21T23:47:34.049556+0000) 2024-02-21T23:47:36.680 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:36 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:36.270+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:05.990749+0000 front 2024-02-21T23:47:05.990685+0000 (oldest deadline 2024-02-21T23:47:32.991590+0000) 2024-02-21T23:47:37.176 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:36 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:36.838+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:47:37.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:36 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:36.975+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:47:37.479 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:37 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:37.156+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:47:37.479 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:37 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:37.255+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:08.122686+0000 front 2024-02-21T23:47:08.122651+0000 (oldest deadline 2024-02-21T23:47:34.022914+0000) 2024-02-21T23:47:37.676 INFO:journalctl@ceph.osd.5.smithi178.stdout:Feb 21 23:47:37 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-5[61968]: 2024-02-21T23:47:37.457+0000 7f9acb6f8700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.048910+0000 front 2024-02-21T23:47:07.048893+0000 (oldest deadline 2024-02-21T23:47:34.049556+0000) 2024-02-21T23:47:37.676 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:37 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:37.233+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:05.990749+0000 front 2024-02-21T23:47:05.990685+0000 (oldest deadline 2024-02-21T23:47:32.991590+0000) 2024-02-21T23:47:38.176 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:37 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:37.856+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:47:38.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:37 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:37.985+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:47:38.479 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:38 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:38.145+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:47:38.479 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:38 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:38.248+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:08.122686+0000 front 2024-02-21T23:47:08.122651+0000 (oldest deadline 2024-02-21T23:47:34.022914+0000) 2024-02-21T23:47:38.676 INFO:journalctl@ceph.osd.5.smithi178.stdout:Feb 21 23:47:38 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-5[61968]: 2024-02-21T23:47:38.479+0000 7f9acb6f8700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.048910+0000 front 2024-02-21T23:47:07.048893+0000 (oldest deadline 2024-02-21T23:47:34.049556+0000) 2024-02-21T23:47:38.676 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:38 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:38.267+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:05.990749+0000 front 2024-02-21T23:47:05.990685+0000 (oldest deadline 2024-02-21T23:47:32.991590+0000) 2024-02-21T23:47:39.176 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:38 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:38.819+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:47:39.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:38 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:38.985+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:47:39.479 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:39 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:39.161+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:47:39.480 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:39 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:39.208+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:08.122686+0000 front 2024-02-21T23:47:08.122651+0000 (oldest deadline 2024-02-21T23:47:34.022914+0000) 2024-02-21T23:47:39.676 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:39 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:39.266+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:05.990749+0000 front 2024-02-21T23:47:05.990685+0000 (oldest deadline 2024-02-21T23:47:32.991590+0000) 2024-02-21T23:47:39.676 INFO:journalctl@ceph.osd.5.smithi178.stdout:Feb 21 23:47:39 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-5[61968]: 2024-02-21T23:47:39.432+0000 7f9acb6f8700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.048910+0000 front 2024-02-21T23:47:07.048893+0000 (oldest deadline 2024-02-21T23:47:34.049556+0000) 2024-02-21T23:47:40.176 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:39 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:39.787+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:47:40.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:39 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:39.787+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:47:40.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:40 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:40.009+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:47:40.479 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:40 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:40.195+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:47:40.480 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:40 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:40.255+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:08.122686+0000 front 2024-02-21T23:47:08.122651+0000 (oldest deadline 2024-02-21T23:47:34.022914+0000) 2024-02-21T23:47:40.480 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:40 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:40.255+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.024574+0000 front 2024-02-21T23:47:15.024594+0000 (oldest deadline 2024-02-21T23:47:39.625031+0000) 2024-02-21T23:47:40.676 INFO:journalctl@ceph.osd.5.smithi178.stdout:Feb 21 23:47:40 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-5[61968]: 2024-02-21T23:47:40.434+0000 7f9acb6f8700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.048910+0000 front 2024-02-21T23:47:07.048893+0000 (oldest deadline 2024-02-21T23:47:34.049556+0000) 2024-02-21T23:47:40.676 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:40 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:40.274+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:05.990749+0000 front 2024-02-21T23:47:05.990685+0000 (oldest deadline 2024-02-21T23:47:32.991590+0000) 2024-02-21T23:47:41.086 INFO:journalctl@ceph.osd.4.smithi176.stdout:Feb 21 23:47:40 smithi176 bash[85549]: ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-4 2024-02-21T23:47:41.087 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:40 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:40.979+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:47:41.176 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:40 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:40.786+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:47:41.176 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:40 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:40.786+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:47:41.479 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:41 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:41.155+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:47:41.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:41 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:41.155+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.583627+0000 front 2024-02-21T23:47:15.583731+0000 (oldest deadline 2024-02-21T23:47:40.883557+0000) 2024-02-21T23:47:41.480 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:41 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:41.283+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:08.122686+0000 front 2024-02-21T23:47:08.122651+0000 (oldest deadline 2024-02-21T23:47:34.022914+0000) 2024-02-21T23:47:41.480 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:41 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:41.283+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.024574+0000 front 2024-02-21T23:47:15.024594+0000 (oldest deadline 2024-02-21T23:47:39.625031+0000) 2024-02-21T23:47:41.586 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:47:41 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:47:41.530+0000 7ff459940700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.307063+0000 front 2024-02-21T23:47:15.307087+0000 (oldest deadline 2024-02-21T23:47:40.607300+0000) 2024-02-21T23:47:41.676 INFO:journalctl@ceph.osd.5.smithi178.stdout:Feb 21 23:47:41 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-5[61968]: 2024-02-21T23:47:41.399+0000 7f9acb6f8700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.048910+0000 front 2024-02-21T23:47:07.048893+0000 (oldest deadline 2024-02-21T23:47:34.049556+0000) 2024-02-21T23:47:41.676 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:41 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:41.290+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:05.990749+0000 front 2024-02-21T23:47:05.990685+0000 (oldest deadline 2024-02-21T23:47:32.991590+0000) 2024-02-21T23:47:41.982 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@osd.4.service' 2024-02-21T23:47:42.024 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-21T23:47:42.024 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-02-21T23:47:42.024 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-02-21T23:47:42.025 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-7ed81218-d112-11ee-95bd-87774f69a715@osd.5 2024-02-21T23:47:42.056 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:41 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:41.784+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:47:42.057 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:41 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:41.784+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:47:42.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:41 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:41.964+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:47:42.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:41 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:41.964+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:47:42.378 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:42 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:42.275+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:05.990749+0000 front 2024-02-21T23:47:05.990685+0000 (oldest deadline 2024-02-21T23:47:32.991590+0000) 2024-02-21T23:47:42.379 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:42 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:42.275+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:18.193453+0000 front 2024-02-21T23:47:18.193604+0000 (oldest deadline 2024-02-21T23:47:41.693472+0000) 2024-02-21T23:47:42.379 INFO:journalctl@ceph.osd.5.smithi178.stdout:Feb 21 23:47:42 smithi178 systemd[1]: Stopping Ceph osd.5 for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:47:42.379 INFO:journalctl@ceph.osd.5.smithi178.stdout:Feb 21 23:47:42 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-5[61968]: 2024-02-21T23:47:42.282+0000 7f9acf903700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-21T23:47:42.379 INFO:journalctl@ceph.osd.5.smithi178.stdout:Feb 21 23:47:42 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-5[61968]: 2024-02-21T23:47:42.282+0000 7f9acf903700 -1 osd.5 85 *** Got signal Terminated *** 2024-02-21T23:47:42.379 INFO:journalctl@ceph.osd.5.smithi178.stdout:Feb 21 23:47:42 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-5[61968]: 2024-02-21T23:47:42.282+0000 7f9acf903700 -1 osd.5 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-21T23:47:42.379 INFO:journalctl@ceph.osd.5.smithi178.stdout:Feb 21 23:47:42 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-5[61968]: 2024-02-21T23:47:42.392+0000 7f9acb6f8700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.048910+0000 front 2024-02-21T23:47:07.048893+0000 (oldest deadline 2024-02-21T23:47:34.049556+0000) 2024-02-21T23:47:42.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:42 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:42.147+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:47:42.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:42 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:42.147+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.583627+0000 front 2024-02-21T23:47:15.583731+0000 (oldest deadline 2024-02-21T23:47:40.883557+0000) 2024-02-21T23:47:42.480 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:42 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:42.240+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:08.122686+0000 front 2024-02-21T23:47:08.122651+0000 (oldest deadline 2024-02-21T23:47:34.022914+0000) 2024-02-21T23:47:42.480 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:42 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:42.240+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.024574+0000 front 2024-02-21T23:47:15.024594+0000 (oldest deadline 2024-02-21T23:47:39.625031+0000) 2024-02-21T23:47:42.836 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:47:42 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:47:42.546+0000 7ff459940700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.307063+0000 front 2024-02-21T23:47:15.307087+0000 (oldest deadline 2024-02-21T23:47:40.607300+0000) 2024-02-21T23:47:43.176 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:42 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:42.824+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:47:43.176 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:42 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:42.824+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:47:43.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:42 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:42.941+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:47:43.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:42 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:42.941+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:47:43.479 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:43 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:43.195+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:47:43.479 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:43 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:43.195+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.583627+0000 front 2024-02-21T23:47:15.583731+0000 (oldest deadline 2024-02-21T23:47:40.883557+0000) 2024-02-21T23:47:43.480 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:43 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:43.248+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:08.122686+0000 front 2024-02-21T23:47:08.122651+0000 (oldest deadline 2024-02-21T23:47:34.022914+0000) 2024-02-21T23:47:43.480 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:43 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:43.248+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.024574+0000 front 2024-02-21T23:47:15.024594+0000 (oldest deadline 2024-02-21T23:47:39.625031+0000) 2024-02-21T23:47:43.677 INFO:journalctl@ceph.osd.5.smithi178.stdout:Feb 21 23:47:43 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-5[61968]: 2024-02-21T23:47:43.409+0000 7f9acb6f8700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.048910+0000 front 2024-02-21T23:47:07.048893+0000 (oldest deadline 2024-02-21T23:47:34.049556+0000) 2024-02-21T23:47:43.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:43 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:43.261+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:05.990749+0000 front 2024-02-21T23:47:05.990685+0000 (oldest deadline 2024-02-21T23:47:32.991590+0000) 2024-02-21T23:47:43.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:43 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:43.261+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:18.193453+0000 front 2024-02-21T23:47:18.193604+0000 (oldest deadline 2024-02-21T23:47:41.693472+0000) 2024-02-21T23:47:43.836 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:47:43 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:47:43.546+0000 7ff459940700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.307063+0000 front 2024-02-21T23:47:15.307087+0000 (oldest deadline 2024-02-21T23:47:40.607300+0000) 2024-02-21T23:47:44.176 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:43 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:43.838+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:47:44.176 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:43 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:43.838+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:47:44.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:43 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:43.962+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:47:44.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:43 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:43.962+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:47:44.479 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:44 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:44.170+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:47:44.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:44 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:44.170+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.583627+0000 front 2024-02-21T23:47:15.583731+0000 (oldest deadline 2024-02-21T23:47:40.883557+0000) 2024-02-21T23:47:44.480 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:44 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:44.266+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:08.122686+0000 front 2024-02-21T23:47:08.122651+0000 (oldest deadline 2024-02-21T23:47:34.022914+0000) 2024-02-21T23:47:44.480 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:44 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:44.266+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.024574+0000 front 2024-02-21T23:47:15.024594+0000 (oldest deadline 2024-02-21T23:47:39.625031+0000) 2024-02-21T23:47:44.676 INFO:journalctl@ceph.osd.5.smithi178.stdout:Feb 21 23:47:44 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-5[61968]: 2024-02-21T23:47:44.402+0000 7f9acb6f8700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.048910+0000 front 2024-02-21T23:47:07.048893+0000 (oldest deadline 2024-02-21T23:47:34.049556+0000) 2024-02-21T23:47:44.676 INFO:journalctl@ceph.osd.5.smithi178.stdout:Feb 21 23:47:44 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-5[61968]: 2024-02-21T23:47:44.402+0000 7f9acb6f8700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:18.051191+0000 front 2024-02-21T23:47:18.051244+0000 (oldest deadline 2024-02-21T23:47:43.951231+0000) 2024-02-21T23:47:44.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:44 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:44.258+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:05.990749+0000 front 2024-02-21T23:47:05.990685+0000 (oldest deadline 2024-02-21T23:47:32.991590+0000) 2024-02-21T23:47:44.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:44 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:44.258+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:18.193453+0000 front 2024-02-21T23:47:18.193604+0000 (oldest deadline 2024-02-21T23:47:41.693472+0000) 2024-02-21T23:47:44.836 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:47:44 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:47:44.564+0000 7ff459940700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.307063+0000 front 2024-02-21T23:47:15.307087+0000 (oldest deadline 2024-02-21T23:47:40.607300+0000) 2024-02-21T23:47:45.176 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:44 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:44.790+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:47:45.176 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:44 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:44.790+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:47:45.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:44 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:44.976+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:47:45.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:44 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:44.976+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:47:45.479 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:45 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:45.140+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:47:45.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:45 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:45.140+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.583627+0000 front 2024-02-21T23:47:15.583731+0000 (oldest deadline 2024-02-21T23:47:40.883557+0000) 2024-02-21T23:47:45.480 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:45 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:45.252+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:08.122686+0000 front 2024-02-21T23:47:08.122651+0000 (oldest deadline 2024-02-21T23:47:34.022914+0000) 2024-02-21T23:47:45.480 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:45 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:45.252+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.024574+0000 front 2024-02-21T23:47:15.024594+0000 (oldest deadline 2024-02-21T23:47:39.625031+0000) 2024-02-21T23:47:45.677 INFO:journalctl@ceph.osd.5.smithi178.stdout:Feb 21 23:47:45 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-5[61968]: 2024-02-21T23:47:45.380+0000 7f9acb6f8700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.048910+0000 front 2024-02-21T23:47:07.048893+0000 (oldest deadline 2024-02-21T23:47:34.049556+0000) 2024-02-21T23:47:45.677 INFO:journalctl@ceph.osd.5.smithi178.stdout:Feb 21 23:47:45 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-5[61968]: 2024-02-21T23:47:45.380+0000 7f9acb6f8700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:18.051191+0000 front 2024-02-21T23:47:18.051244+0000 (oldest deadline 2024-02-21T23:47:43.951231+0000) 2024-02-21T23:47:45.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:45 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:45.250+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:05.990749+0000 front 2024-02-21T23:47:05.990685+0000 (oldest deadline 2024-02-21T23:47:32.991590+0000) 2024-02-21T23:47:45.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:45 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:45.250+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:18.193453+0000 front 2024-02-21T23:47:18.193604+0000 (oldest deadline 2024-02-21T23:47:41.693472+0000) 2024-02-21T23:47:45.836 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:47:45 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:47:45.557+0000 7ff459940700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.307063+0000 front 2024-02-21T23:47:15.307087+0000 (oldest deadline 2024-02-21T23:47:40.607300+0000) 2024-02-21T23:47:46.176 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:45 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:45.829+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:47:46.176 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:45 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:45.829+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:47:46.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:45 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:45.972+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:47:46.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:45 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:45.972+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:47:46.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:46 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:46.123+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:47:46.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:46 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:46.123+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.583627+0000 front 2024-02-21T23:47:15.583731+0000 (oldest deadline 2024-02-21T23:47:40.883557+0000) 2024-02-21T23:47:46.480 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:46 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:46.248+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:08.122686+0000 front 2024-02-21T23:47:08.122651+0000 (oldest deadline 2024-02-21T23:47:34.022914+0000) 2024-02-21T23:47:46.480 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:46 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:46.248+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.024574+0000 front 2024-02-21T23:47:15.024594+0000 (oldest deadline 2024-02-21T23:47:39.625031+0000) 2024-02-21T23:47:46.676 INFO:journalctl@ceph.osd.5.smithi178.stdout:Feb 21 23:47:46 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-5[61968]: 2024-02-21T23:47:46.364+0000 7f9acb6f8700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.048910+0000 front 2024-02-21T23:47:07.048893+0000 (oldest deadline 2024-02-21T23:47:34.049556+0000) 2024-02-21T23:47:46.677 INFO:journalctl@ceph.osd.5.smithi178.stdout:Feb 21 23:47:46 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-5[61968]: 2024-02-21T23:47:46.364+0000 7f9acb6f8700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:18.051191+0000 front 2024-02-21T23:47:18.051244+0000 (oldest deadline 2024-02-21T23:47:43.951231+0000) 2024-02-21T23:47:46.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:46 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:46.274+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:05.990749+0000 front 2024-02-21T23:47:05.990685+0000 (oldest deadline 2024-02-21T23:47:32.991590+0000) 2024-02-21T23:47:46.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:46 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:46.274+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:18.193453+0000 front 2024-02-21T23:47:18.193604+0000 (oldest deadline 2024-02-21T23:47:41.693472+0000) 2024-02-21T23:47:46.836 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:47:46 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:47:46.538+0000 7ff459940700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.307063+0000 front 2024-02-21T23:47:15.307087+0000 (oldest deadline 2024-02-21T23:47:40.607300+0000) 2024-02-21T23:47:47.176 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:46 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:46.799+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:47:47.176 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:46 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:46.799+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:47:47.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:47 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:47.004+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:47:47.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:47 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:47.004+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:47:47.479 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:47 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:47.149+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:47:47.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:47 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:47.149+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.583627+0000 front 2024-02-21T23:47:15.583731+0000 (oldest deadline 2024-02-21T23:47:40.883557+0000) 2024-02-21T23:47:47.480 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:47 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:47.275+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:08.122686+0000 front 2024-02-21T23:47:08.122651+0000 (oldest deadline 2024-02-21T23:47:34.022914+0000) 2024-02-21T23:47:47.480 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:47 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:47.275+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.024574+0000 front 2024-02-21T23:47:15.024594+0000 (oldest deadline 2024-02-21T23:47:39.625031+0000) 2024-02-21T23:47:47.679 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:47 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:47.310+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:05.990749+0000 front 2024-02-21T23:47:05.990685+0000 (oldest deadline 2024-02-21T23:47:32.991590+0000) 2024-02-21T23:47:47.679 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:47 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:47.310+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:18.193453+0000 front 2024-02-21T23:47:18.193604+0000 (oldest deadline 2024-02-21T23:47:41.693472+0000) 2024-02-21T23:47:47.679 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:47 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:47.310+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:23.394442+0000 front 2024-02-21T23:47:23.394464+0000 (oldest deadline 2024-02-21T23:47:46.894536+0000) 2024-02-21T23:47:47.836 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:47:47 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:47:47.535+0000 7ff459940700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.307063+0000 front 2024-02-21T23:47:15.307087+0000 (oldest deadline 2024-02-21T23:47:40.607300+0000) 2024-02-21T23:47:48.177 INFO:journalctl@ceph.osd.5.smithi178.stdout:Feb 21 23:47:47 smithi178 bash[87633]: ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-5 2024-02-21T23:47:48.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:47 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:47.826+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:47:48.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:47 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:47.826+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:47:48.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:48 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:48.000+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:47:48.339 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:48 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:48.000+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:47:48.480 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:48 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:48.249+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:08.122686+0000 front 2024-02-21T23:47:08.122651+0000 (oldest deadline 2024-02-21T23:47:34.022914+0000) 2024-02-21T23:47:48.480 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:48 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:48.249+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.024574+0000 front 2024-02-21T23:47:15.024594+0000 (oldest deadline 2024-02-21T23:47:39.625031+0000) 2024-02-21T23:47:48.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:48 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:48.111+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:47:48.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:48 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:48.111+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.583627+0000 front 2024-02-21T23:47:15.583731+0000 (oldest deadline 2024-02-21T23:47:40.883557+0000) 2024-02-21T23:47:48.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:48 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:48.111+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:21.384659+0000 front 2024-02-21T23:47:21.384611+0000 (oldest deadline 2024-02-21T23:47:47.284798+0000) 2024-02-21T23:47:48.676 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:48 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:48.314+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:05.990749+0000 front 2024-02-21T23:47:05.990685+0000 (oldest deadline 2024-02-21T23:47:32.991590+0000) 2024-02-21T23:47:48.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:48 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:48.314+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:18.193453+0000 front 2024-02-21T23:47:18.193604+0000 (oldest deadline 2024-02-21T23:47:41.693472+0000) 2024-02-21T23:47:48.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:48 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:48.314+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:23.394442+0000 front 2024-02-21T23:47:23.394464+0000 (oldest deadline 2024-02-21T23:47:46.894536+0000) 2024-02-21T23:47:48.836 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:47:48 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:47:48.567+0000 7ff459940700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.307063+0000 front 2024-02-21T23:47:15.307087+0000 (oldest deadline 2024-02-21T23:47:40.607300+0000) 2024-02-21T23:47:49.159 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:48 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:48.868+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:47:49.159 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:48 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:48.868+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:47:49.164 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@osd.5.service' 2024-02-21T23:47:49.245 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-21T23:47:49.246 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-02-21T23:47:49.246 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-02-21T23:47:49.246 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-7ed81218-d112-11ee-95bd-87774f69a715@osd.6 2024-02-21T23:47:49.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:49 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:49.005+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:47:49.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:49 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:49.005+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:47:49.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:49 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:49.005+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.160285+0000 front 2024-02-21T23:47:24.160271+0000 (oldest deadline 2024-02-21T23:47:48.860330+0000) 2024-02-21T23:47:49.426 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:49 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:49.309+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:05.990749+0000 front 2024-02-21T23:47:05.990685+0000 (oldest deadline 2024-02-21T23:47:32.991590+0000) 2024-02-21T23:47:49.426 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:49 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:49.309+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:18.193453+0000 front 2024-02-21T23:47:18.193604+0000 (oldest deadline 2024-02-21T23:47:41.693472+0000) 2024-02-21T23:47:49.427 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:49 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:49.309+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:23.394442+0000 front 2024-02-21T23:47:23.394464+0000 (oldest deadline 2024-02-21T23:47:46.894536+0000) 2024-02-21T23:47:49.473 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:49 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:49.259+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:08.122686+0000 front 2024-02-21T23:47:08.122651+0000 (oldest deadline 2024-02-21T23:47:34.022914+0000) 2024-02-21T23:47:49.473 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:49 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:49.259+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.024574+0000 front 2024-02-21T23:47:15.024594+0000 (oldest deadline 2024-02-21T23:47:39.625031+0000) 2024-02-21T23:47:49.474 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:49 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:49.259+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.326077+0000 front 2024-02-21T23:47:24.326194+0000 (oldest deadline 2024-02-21T23:47:48.426023+0000) 2024-02-21T23:47:49.474 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:49 smithi090 systemd[1]: Stopping Ceph osd.6 for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:47:49.474 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:49 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:49.486+0000 7fab50b76700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-21T23:47:49.474 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:49 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:49.486+0000 7fab50b76700 -1 osd.6 85 *** Got signal Terminated *** 2024-02-21T23:47:49.474 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:49 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:49.082+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:47:49.474 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:49 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:49.082+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.583627+0000 front 2024-02-21T23:47:15.583731+0000 (oldest deadline 2024-02-21T23:47:40.883557+0000) 2024-02-21T23:47:49.474 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:49 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:49.082+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:21.384659+0000 front 2024-02-21T23:47:21.384611+0000 (oldest deadline 2024-02-21T23:47:47.284798+0000) 2024-02-21T23:47:49.729 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:49 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:49.486+0000 7fab50b76700 -1 osd.6 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-21T23:47:49.836 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:47:49 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:47:49.578+0000 7ff459940700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.307063+0000 front 2024-02-21T23:47:15.307087+0000 (oldest deadline 2024-02-21T23:47:40.607300+0000) 2024-02-21T23:47:50.176 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:49 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:49.826+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:47:50.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:49 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:49.826+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:47:50.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:50 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:50.019+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:47:50.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:50 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:50.019+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:47:50.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:50 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:50.019+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.160285+0000 front 2024-02-21T23:47:24.160271+0000 (oldest deadline 2024-02-21T23:47:48.860330+0000) 2024-02-21T23:47:50.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:50 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:50.073+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:47:50.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:50 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:50.073+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.583627+0000 front 2024-02-21T23:47:15.583731+0000 (oldest deadline 2024-02-21T23:47:40.883557+0000) 2024-02-21T23:47:50.481 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:50 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:50.073+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:21.384659+0000 front 2024-02-21T23:47:21.384611+0000 (oldest deadline 2024-02-21T23:47:47.284798+0000) 2024-02-21T23:47:50.481 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:50 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:50.296+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:08.122686+0000 front 2024-02-21T23:47:08.122651+0000 (oldest deadline 2024-02-21T23:47:34.022914+0000) 2024-02-21T23:47:50.481 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:50 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:50.296+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.024574+0000 front 2024-02-21T23:47:15.024594+0000 (oldest deadline 2024-02-21T23:47:39.625031+0000) 2024-02-21T23:47:50.481 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:50 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:50.296+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.326077+0000 front 2024-02-21T23:47:24.326194+0000 (oldest deadline 2024-02-21T23:47:48.426023+0000) 2024-02-21T23:47:50.676 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:50 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:50.317+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:05.990749+0000 front 2024-02-21T23:47:05.990685+0000 (oldest deadline 2024-02-21T23:47:32.991590+0000) 2024-02-21T23:47:50.676 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:50 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:50.317+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:18.193453+0000 front 2024-02-21T23:47:18.193604+0000 (oldest deadline 2024-02-21T23:47:41.693472+0000) 2024-02-21T23:47:50.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:50 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:50.317+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:23.394442+0000 front 2024-02-21T23:47:23.394464+0000 (oldest deadline 2024-02-21T23:47:46.894536+0000) 2024-02-21T23:47:50.836 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:47:50 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:47:50.584+0000 7ff459940700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.307063+0000 front 2024-02-21T23:47:15.307087+0000 (oldest deadline 2024-02-21T23:47:40.607300+0000) 2024-02-21T23:47:50.836 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:47:50 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:47:50.584+0000 7ff459940700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.108705+0000 front 2024-02-21T23:47:24.108718+0000 (oldest deadline 2024-02-21T23:47:50.008543+0000) 2024-02-21T23:47:51.176 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:50 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:50.863+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:47:51.176 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:50 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:50.863+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:47:51.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:50 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:50.863+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:47:51.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:51 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:51.049+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:47:51.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:51 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:51.049+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:47:51.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:51 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:51.049+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.160285+0000 front 2024-02-21T23:47:24.160271+0000 (oldest deadline 2024-02-21T23:47:48.860330+0000) 2024-02-21T23:47:51.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:51 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:51.111+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:47:51.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:51 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:51.111+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.583627+0000 front 2024-02-21T23:47:15.583731+0000 (oldest deadline 2024-02-21T23:47:40.883557+0000) 2024-02-21T23:47:51.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:51 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:51.111+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:21.384659+0000 front 2024-02-21T23:47:21.384611+0000 (oldest deadline 2024-02-21T23:47:47.284798+0000) 2024-02-21T23:47:51.480 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:51 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:51.256+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:08.122686+0000 front 2024-02-21T23:47:08.122651+0000 (oldest deadline 2024-02-21T23:47:34.022914+0000) 2024-02-21T23:47:51.481 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:51 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:51.256+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.024574+0000 front 2024-02-21T23:47:15.024594+0000 (oldest deadline 2024-02-21T23:47:39.625031+0000) 2024-02-21T23:47:51.481 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:51 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:51.256+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.326077+0000 front 2024-02-21T23:47:24.326194+0000 (oldest deadline 2024-02-21T23:47:48.426023+0000) 2024-02-21T23:47:51.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:51 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:51.344+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:05.990749+0000 front 2024-02-21T23:47:05.990685+0000 (oldest deadline 2024-02-21T23:47:32.991590+0000) 2024-02-21T23:47:51.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:51 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:51.344+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:18.193453+0000 front 2024-02-21T23:47:18.193604+0000 (oldest deadline 2024-02-21T23:47:41.693472+0000) 2024-02-21T23:47:51.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:51 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:51.344+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:23.394442+0000 front 2024-02-21T23:47:23.394464+0000 (oldest deadline 2024-02-21T23:47:46.894536+0000) 2024-02-21T23:47:51.836 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:47:51 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:47:51.589+0000 7ff459940700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.307063+0000 front 2024-02-21T23:47:15.307087+0000 (oldest deadline 2024-02-21T23:47:40.607300+0000) 2024-02-21T23:47:51.836 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:47:51 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:47:51.589+0000 7ff459940700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.108705+0000 front 2024-02-21T23:47:24.108718+0000 (oldest deadline 2024-02-21T23:47:50.008543+0000) 2024-02-21T23:47:52.176 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:51 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:51.910+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:47:52.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:51 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:51.910+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:47:52.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:51 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:51.910+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:47:52.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:52 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:52.002+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:47:52.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:52 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:52.002+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:47:52.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:52 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:52.002+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.160285+0000 front 2024-02-21T23:47:24.160271+0000 (oldest deadline 2024-02-21T23:47:48.860330+0000) 2024-02-21T23:47:52.479 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:52 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:52.131+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:47:52.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:52 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:52.131+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.583627+0000 front 2024-02-21T23:47:15.583731+0000 (oldest deadline 2024-02-21T23:47:40.883557+0000) 2024-02-21T23:47:52.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:52 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:52.131+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:21.384659+0000 front 2024-02-21T23:47:21.384611+0000 (oldest deadline 2024-02-21T23:47:47.284798+0000) 2024-02-21T23:47:52.480 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:52 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:52.219+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:08.122686+0000 front 2024-02-21T23:47:08.122651+0000 (oldest deadline 2024-02-21T23:47:34.022914+0000) 2024-02-21T23:47:52.480 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:52 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:52.219+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.024574+0000 front 2024-02-21T23:47:15.024594+0000 (oldest deadline 2024-02-21T23:47:39.625031+0000) 2024-02-21T23:47:52.480 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:52 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:52.219+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.326077+0000 front 2024-02-21T23:47:24.326194+0000 (oldest deadline 2024-02-21T23:47:48.426023+0000) 2024-02-21T23:47:52.676 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:52 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:52.376+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:05.990749+0000 front 2024-02-21T23:47:05.990685+0000 (oldest deadline 2024-02-21T23:47:32.991590+0000) 2024-02-21T23:47:52.676 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:52 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:52.376+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:18.193453+0000 front 2024-02-21T23:47:18.193604+0000 (oldest deadline 2024-02-21T23:47:41.693472+0000) 2024-02-21T23:47:52.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:52 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:52.376+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:23.394442+0000 front 2024-02-21T23:47:23.394464+0000 (oldest deadline 2024-02-21T23:47:46.894536+0000) 2024-02-21T23:47:52.990 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:47:52 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:47:52.604+0000 7ff459940700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.307063+0000 front 2024-02-21T23:47:15.307087+0000 (oldest deadline 2024-02-21T23:47:40.607300+0000) 2024-02-21T23:47:52.991 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:47:52 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:47:52.604+0000 7ff459940700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.108705+0000 front 2024-02-21T23:47:24.108718+0000 (oldest deadline 2024-02-21T23:47:50.008543+0000) 2024-02-21T23:47:53.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:52 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:52.909+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:47:53.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:52 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:52.909+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:47:53.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:52 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:52.909+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:47:53.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:53 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:53.003+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:47:53.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:53 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:53.003+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:47:53.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:53 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:53.003+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.160285+0000 front 2024-02-21T23:47:24.160271+0000 (oldest deadline 2024-02-21T23:47:48.860330+0000) 2024-02-21T23:47:53.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:53 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:53.173+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:47:53.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:53 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:53.173+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.583627+0000 front 2024-02-21T23:47:15.583731+0000 (oldest deadline 2024-02-21T23:47:40.883557+0000) 2024-02-21T23:47:53.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:53 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:53.173+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:21.384659+0000 front 2024-02-21T23:47:21.384611+0000 (oldest deadline 2024-02-21T23:47:47.284798+0000) 2024-02-21T23:47:53.480 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:53 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:53.233+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:08.122686+0000 front 2024-02-21T23:47:08.122651+0000 (oldest deadline 2024-02-21T23:47:34.022914+0000) 2024-02-21T23:47:53.480 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:53 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:53.233+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.024574+0000 front 2024-02-21T23:47:15.024594+0000 (oldest deadline 2024-02-21T23:47:39.625031+0000) 2024-02-21T23:47:53.480 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:53 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:53.234+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.326077+0000 front 2024-02-21T23:47:24.326194+0000 (oldest deadline 2024-02-21T23:47:48.426023+0000) 2024-02-21T23:47:53.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:53 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:53.400+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:05.990749+0000 front 2024-02-21T23:47:05.990685+0000 (oldest deadline 2024-02-21T23:47:32.991590+0000) 2024-02-21T23:47:53.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:53 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:53.401+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:18.193453+0000 front 2024-02-21T23:47:18.193604+0000 (oldest deadline 2024-02-21T23:47:41.693472+0000) 2024-02-21T23:47:53.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:53 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:53.401+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:23.394442+0000 front 2024-02-21T23:47:23.394464+0000 (oldest deadline 2024-02-21T23:47:46.894536+0000) 2024-02-21T23:47:54.031 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:47:53 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:47:53.654+0000 7ff459940700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.307063+0000 front 2024-02-21T23:47:15.307087+0000 (oldest deadline 2024-02-21T23:47:40.607300+0000) 2024-02-21T23:47:54.031 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:47:53 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:47:53.654+0000 7ff459940700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.108705+0000 front 2024-02-21T23:47:24.108718+0000 (oldest deadline 2024-02-21T23:47:50.008543+0000) 2024-02-21T23:47:54.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:53 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:53.897+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:47:54.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:53 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:53.897+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:47:54.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:53 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:53.897+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:47:54.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:54 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:54.044+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:47:54.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:54 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:54.044+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:47:54.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:54 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:54.044+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.160285+0000 front 2024-02-21T23:47:24.160271+0000 (oldest deadline 2024-02-21T23:47:48.860330+0000) 2024-02-21T23:47:54.480 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:54 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:54.187+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:08.122686+0000 front 2024-02-21T23:47:08.122651+0000 (oldest deadline 2024-02-21T23:47:34.022914+0000) 2024-02-21T23:47:54.480 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:54 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:54.187+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.024574+0000 front 2024-02-21T23:47:15.024594+0000 (oldest deadline 2024-02-21T23:47:39.625031+0000) 2024-02-21T23:47:54.480 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:54 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6[78522]: 2024-02-21T23:47:54.187+0000 7fab49965700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.326077+0000 front 2024-02-21T23:47:24.326194+0000 (oldest deadline 2024-02-21T23:47:48.426023+0000) 2024-02-21T23:47:54.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:54 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:54.218+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:47:54.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:54 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:54.218+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.583627+0000 front 2024-02-21T23:47:15.583731+0000 (oldest deadline 2024-02-21T23:47:40.883557+0000) 2024-02-21T23:47:54.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:54 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:54.218+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:21.384659+0000 front 2024-02-21T23:47:21.384611+0000 (oldest deadline 2024-02-21T23:47:47.284798+0000) 2024-02-21T23:47:54.676 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:54 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:54.387+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:05.990749+0000 front 2024-02-21T23:47:05.990685+0000 (oldest deadline 2024-02-21T23:47:32.991590+0000) 2024-02-21T23:47:54.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:54 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:54.387+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:18.193453+0000 front 2024-02-21T23:47:18.193604+0000 (oldest deadline 2024-02-21T23:47:41.693472+0000) 2024-02-21T23:47:54.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:54 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:54.387+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:23.394442+0000 front 2024-02-21T23:47:23.394464+0000 (oldest deadline 2024-02-21T23:47:46.894536+0000) 2024-02-21T23:47:55.062 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:47:54 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:47:54.606+0000 7ff459940700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.307063+0000 front 2024-02-21T23:47:15.307087+0000 (oldest deadline 2024-02-21T23:47:40.607300+0000) 2024-02-21T23:47:55.062 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:47:54 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:47:54.606+0000 7ff459940700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.108705+0000 front 2024-02-21T23:47:24.108718+0000 (oldest deadline 2024-02-21T23:47:50.008543+0000) 2024-02-21T23:47:55.176 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:54 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:54.911+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:47:55.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:54 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:54.911+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:47:55.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:54 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:54.911+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:47:55.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:55 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:55.075+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:47:55.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:55 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:55.075+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:47:55.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:55 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:55.075+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.160285+0000 front 2024-02-21T23:47:24.160271+0000 (oldest deadline 2024-02-21T23:47:48.860330+0000) 2024-02-21T23:47:55.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:55 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:55.075+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:32.361609+0000 front 2024-02-21T23:47:32.361527+0000 (oldest deadline 2024-02-21T23:47:54.661354+0000) 2024-02-21T23:47:55.480 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 21 23:47:55 smithi090 bash[99878]: ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-6 2024-02-21T23:47:55.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:55 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:55.174+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:47:55.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:55 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:55.174+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.583627+0000 front 2024-02-21T23:47:15.583731+0000 (oldest deadline 2024-02-21T23:47:40.883557+0000) 2024-02-21T23:47:55.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:55 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:55.174+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:21.384659+0000 front 2024-02-21T23:47:21.384611+0000 (oldest deadline 2024-02-21T23:47:47.284798+0000) 2024-02-21T23:47:55.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:55 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:55.399+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:05.990749+0000 front 2024-02-21T23:47:05.990685+0000 (oldest deadline 2024-02-21T23:47:32.991590+0000) 2024-02-21T23:47:55.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:55 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:55.399+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:18.193453+0000 front 2024-02-21T23:47:18.193604+0000 (oldest deadline 2024-02-21T23:47:41.693472+0000) 2024-02-21T23:47:55.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:55 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:55.399+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:23.394442+0000 front 2024-02-21T23:47:23.394464+0000 (oldest deadline 2024-02-21T23:47:46.894536+0000) 2024-02-21T23:47:56.067 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:47:55 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:47:55.617+0000 7ff459940700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.307063+0000 front 2024-02-21T23:47:15.307087+0000 (oldest deadline 2024-02-21T23:47:40.607300+0000) 2024-02-21T23:47:56.067 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:47:55 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:47:55.617+0000 7ff459940700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.108705+0000 front 2024-02-21T23:47:24.108718+0000 (oldest deadline 2024-02-21T23:47:50.008543+0000) 2024-02-21T23:47:56.176 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:55 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:55.897+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:47:56.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:55 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:55.897+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:47:56.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:55 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:55.897+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:47:56.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:56 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:56.080+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:47:56.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:56 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:56.080+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:47:56.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:56 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:56.080+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.160285+0000 front 2024-02-21T23:47:24.160271+0000 (oldest deadline 2024-02-21T23:47:48.860330+0000) 2024-02-21T23:47:56.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:56 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:56.080+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:32.361609+0000 front 2024-02-21T23:47:32.361527+0000 (oldest deadline 2024-02-21T23:47:54.661354+0000) 2024-02-21T23:47:56.427 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:56 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:56.128+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:47:56.428 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:56 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:56.128+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.583627+0000 front 2024-02-21T23:47:15.583731+0000 (oldest deadline 2024-02-21T23:47:40.883557+0000) 2024-02-21T23:47:56.428 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:56 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:56.128+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:21.384659+0000 front 2024-02-21T23:47:21.384611+0000 (oldest deadline 2024-02-21T23:47:47.284798+0000) 2024-02-21T23:47:56.428 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:56 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:56.128+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:30.785587+0000 front 2024-02-21T23:47:30.785598+0000 (oldest deadline 2024-02-21T23:47:55.485770+0000) 2024-02-21T23:47:56.434 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@osd.6.service' 2024-02-21T23:47:56.475 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-21T23:47:56.475 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-02-21T23:47:56.475 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-02-21T23:47:56.475 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-7ed81218-d112-11ee-95bd-87774f69a715@osd.7 2024-02-21T23:47:56.676 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:56 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:56.400+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:05.990749+0000 front 2024-02-21T23:47:05.990685+0000 (oldest deadline 2024-02-21T23:47:32.991590+0000) 2024-02-21T23:47:56.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:56 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:56.400+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:18.193453+0000 front 2024-02-21T23:47:18.193604+0000 (oldest deadline 2024-02-21T23:47:41.693472+0000) 2024-02-21T23:47:56.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:56 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:56.400+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:23.394442+0000 front 2024-02-21T23:47:23.394464+0000 (oldest deadline 2024-02-21T23:47:46.894536+0000) 2024-02-21T23:47:56.837 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:47:56 smithi176 systemd[1]: Stopping Ceph osd.7 for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:47:56.837 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:47:56 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:47:56.655+0000 7ff459940700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.307063+0000 front 2024-02-21T23:47:15.307087+0000 (oldest deadline 2024-02-21T23:47:40.607300+0000) 2024-02-21T23:47:56.837 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:47:56 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:47:56.655+0000 7ff459940700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.108705+0000 front 2024-02-21T23:47:24.108718+0000 (oldest deadline 2024-02-21T23:47:50.008543+0000) 2024-02-21T23:47:56.837 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:47:56 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:47:56.731+0000 7ff45f34e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-21T23:47:56.837 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:47:56 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:47:56.731+0000 7ff45f34e700 -1 osd.7 85 *** Got signal Terminated *** 2024-02-21T23:47:56.837 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:47:56 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:47:56.731+0000 7ff45f34e700 -1 osd.7 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-21T23:47:57.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:56 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:56.865+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:47:57.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:56 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:56.865+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:47:57.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:56 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:56.865+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:47:57.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:57 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:57.077+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:47:57.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:57 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:57.077+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:47:57.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:57 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:57.077+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.160285+0000 front 2024-02-21T23:47:24.160271+0000 (oldest deadline 2024-02-21T23:47:48.860330+0000) 2024-02-21T23:47:57.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:57 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:57.077+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:32.361609+0000 front 2024-02-21T23:47:32.361527+0000 (oldest deadline 2024-02-21T23:47:54.661354+0000) 2024-02-21T23:47:57.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:57 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:57.125+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:47:57.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:57 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:57.125+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.583627+0000 front 2024-02-21T23:47:15.583731+0000 (oldest deadline 2024-02-21T23:47:40.883557+0000) 2024-02-21T23:47:57.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:57 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:57.125+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:21.384659+0000 front 2024-02-21T23:47:21.384611+0000 (oldest deadline 2024-02-21T23:47:47.284798+0000) 2024-02-21T23:47:57.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:57 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:57.125+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:30.785587+0000 front 2024-02-21T23:47:30.785598+0000 (oldest deadline 2024-02-21T23:47:55.485770+0000) 2024-02-21T23:47:57.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:57 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:57.398+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:05.990749+0000 front 2024-02-21T23:47:05.990685+0000 (oldest deadline 2024-02-21T23:47:32.991590+0000) 2024-02-21T23:47:57.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:57 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:57.398+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:18.193453+0000 front 2024-02-21T23:47:18.193604+0000 (oldest deadline 2024-02-21T23:47:41.693472+0000) 2024-02-21T23:47:57.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:57 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:57.398+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:23.394442+0000 front 2024-02-21T23:47:23.394464+0000 (oldest deadline 2024-02-21T23:47:46.894536+0000) 2024-02-21T23:47:58.057 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:47:57 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:47:57.659+0000 7ff459940700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.307063+0000 front 2024-02-21T23:47:15.307087+0000 (oldest deadline 2024-02-21T23:47:40.607300+0000) 2024-02-21T23:47:58.057 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:47:57 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:47:57.659+0000 7ff459940700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.108705+0000 front 2024-02-21T23:47:24.108718+0000 (oldest deadline 2024-02-21T23:47:50.008543+0000) 2024-02-21T23:47:58.057 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:47:57 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:47:57.659+0000 7ff459940700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:30.009006+0000 front 2024-02-21T23:47:30.009050+0000 (oldest deadline 2024-02-21T23:47:57.009526+0000) 2024-02-21T23:47:58.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:57 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:57.880+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:47:58.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:57 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:57.880+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:47:58.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:57 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:57.880+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:47:58.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:58 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:58.070+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:47:58.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:58 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:58.070+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:47:58.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:58 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:58.070+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.160285+0000 front 2024-02-21T23:47:24.160271+0000 (oldest deadline 2024-02-21T23:47:48.860330+0000) 2024-02-21T23:47:58.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:58 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:58.070+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:32.361609+0000 front 2024-02-21T23:47:32.361527+0000 (oldest deadline 2024-02-21T23:47:54.661354+0000) 2024-02-21T23:47:58.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:58 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:58.118+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:47:58.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:58 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:58.118+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.583627+0000 front 2024-02-21T23:47:15.583731+0000 (oldest deadline 2024-02-21T23:47:40.883557+0000) 2024-02-21T23:47:58.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:58 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:58.118+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:21.384659+0000 front 2024-02-21T23:47:21.384611+0000 (oldest deadline 2024-02-21T23:47:47.284798+0000) 2024-02-21T23:47:58.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:58 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:58.118+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:30.785587+0000 front 2024-02-21T23:47:30.785598+0000 (oldest deadline 2024-02-21T23:47:55.485770+0000) 2024-02-21T23:47:58.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:58 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:58.400+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:05.990749+0000 front 2024-02-21T23:47:05.990685+0000 (oldest deadline 2024-02-21T23:47:32.991590+0000) 2024-02-21T23:47:58.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:58 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:58.400+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:18.193453+0000 front 2024-02-21T23:47:18.193604+0000 (oldest deadline 2024-02-21T23:47:41.693472+0000) 2024-02-21T23:47:58.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:58 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:58.400+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:23.394442+0000 front 2024-02-21T23:47:23.394464+0000 (oldest deadline 2024-02-21T23:47:46.894536+0000) 2024-02-21T23:47:58.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:58 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:58.400+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:31.595665+0000 front 2024-02-21T23:47:31.595563+0000 (oldest deadline 2024-02-21T23:47:57.495447+0000) 2024-02-21T23:47:59.082 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:47:58 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:47:58.675+0000 7ff459940700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.307063+0000 front 2024-02-21T23:47:15.307087+0000 (oldest deadline 2024-02-21T23:47:40.607300+0000) 2024-02-21T23:47:59.176 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:47:58 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:47:58.675+0000 7ff459940700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.108705+0000 front 2024-02-21T23:47:24.108718+0000 (oldest deadline 2024-02-21T23:47:50.008543+0000) 2024-02-21T23:47:59.176 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:47:58 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:47:58.675+0000 7ff459940700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:30.009006+0000 front 2024-02-21T23:47:30.009050+0000 (oldest deadline 2024-02-21T23:47:57.009526+0000) 2024-02-21T23:47:59.176 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:58 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:58.850+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:47:59.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:58 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:58.850+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:47:59.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:58 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:58.850+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:47:59.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:59 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:59.095+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:47:59.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:59 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:59.095+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:47:59.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:59 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:59.095+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.160285+0000 front 2024-02-21T23:47:24.160271+0000 (oldest deadline 2024-02-21T23:47:48.860330+0000) 2024-02-21T23:47:59.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:47:59 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:47:59.095+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:32.361609+0000 front 2024-02-21T23:47:32.361527+0000 (oldest deadline 2024-02-21T23:47:54.661354+0000) 2024-02-21T23:47:59.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:59 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:59.164+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:47:59.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:59 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:59.164+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.583627+0000 front 2024-02-21T23:47:15.583731+0000 (oldest deadline 2024-02-21T23:47:40.883557+0000) 2024-02-21T23:47:59.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:59 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:59.164+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:21.384659+0000 front 2024-02-21T23:47:21.384611+0000 (oldest deadline 2024-02-21T23:47:47.284798+0000) 2024-02-21T23:47:59.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:47:59 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:47:59.164+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:30.785587+0000 front 2024-02-21T23:47:30.785598+0000 (oldest deadline 2024-02-21T23:47:55.485770+0000) 2024-02-21T23:47:59.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:59 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:59.357+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:05.990749+0000 front 2024-02-21T23:47:05.990685+0000 (oldest deadline 2024-02-21T23:47:32.991590+0000) 2024-02-21T23:47:59.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:59 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:59.357+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:18.193453+0000 front 2024-02-21T23:47:18.193604+0000 (oldest deadline 2024-02-21T23:47:41.693472+0000) 2024-02-21T23:47:59.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:59 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:59.357+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:23.394442+0000 front 2024-02-21T23:47:23.394464+0000 (oldest deadline 2024-02-21T23:47:46.894536+0000) 2024-02-21T23:47:59.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:47:59 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:47:59.357+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:31.595665+0000 front 2024-02-21T23:47:31.595563+0000 (oldest deadline 2024-02-21T23:47:57.495447+0000) 2024-02-21T23:48:00.086 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:47:59 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:47:59.723+0000 7ff459940700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.307063+0000 front 2024-02-21T23:47:15.307087+0000 (oldest deadline 2024-02-21T23:47:40.607300+0000) 2024-02-21T23:48:00.087 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:47:59 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:47:59.723+0000 7ff459940700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.108705+0000 front 2024-02-21T23:47:24.108718+0000 (oldest deadline 2024-02-21T23:47:50.008543+0000) 2024-02-21T23:48:00.087 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:47:59 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:47:59.723+0000 7ff459940700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:30.009006+0000 front 2024-02-21T23:47:30.009050+0000 (oldest deadline 2024-02-21T23:47:57.009526+0000) 2024-02-21T23:48:00.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:59 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:59.892+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:48:00.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:59 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:59.892+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:48:00.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:47:59 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:47:59.892+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:48:00.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:00 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:00.160+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:48:00.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:00 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:00.160+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.583627+0000 front 2024-02-21T23:47:15.583731+0000 (oldest deadline 2024-02-21T23:47:40.883557+0000) 2024-02-21T23:48:00.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:00 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:00.160+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:21.384659+0000 front 2024-02-21T23:47:21.384611+0000 (oldest deadline 2024-02-21T23:47:47.284798+0000) 2024-02-21T23:48:00.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:00 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:00.160+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:30.785587+0000 front 2024-02-21T23:47:30.785598+0000 (oldest deadline 2024-02-21T23:47:55.485770+0000) 2024-02-21T23:48:00.587 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:00 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:00.119+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:48:00.587 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:00 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:00.119+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:48:00.587 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:00 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:00.119+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.160285+0000 front 2024-02-21T23:47:24.160271+0000 (oldest deadline 2024-02-21T23:47:48.860330+0000) 2024-02-21T23:48:00.587 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:00 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:00.119+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:32.361609+0000 front 2024-02-21T23:47:32.361527+0000 (oldest deadline 2024-02-21T23:47:54.661354+0000) 2024-02-21T23:48:00.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:00 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:00.371+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:05.990749+0000 front 2024-02-21T23:47:05.990685+0000 (oldest deadline 2024-02-21T23:47:32.991590+0000) 2024-02-21T23:48:00.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:00 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:00.371+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:18.193453+0000 front 2024-02-21T23:47:18.193604+0000 (oldest deadline 2024-02-21T23:47:41.693472+0000) 2024-02-21T23:48:00.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:00 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:00.371+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:23.394442+0000 front 2024-02-21T23:47:23.394464+0000 (oldest deadline 2024-02-21T23:47:46.894536+0000) 2024-02-21T23:48:00.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:00 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:00.371+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:31.595665+0000 front 2024-02-21T23:47:31.595563+0000 (oldest deadline 2024-02-21T23:47:57.495447+0000) 2024-02-21T23:48:01.080 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:48:00 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:48:00.709+0000 7ff459940700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.307063+0000 front 2024-02-21T23:47:15.307087+0000 (oldest deadline 2024-02-21T23:47:40.607300+0000) 2024-02-21T23:48:01.081 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:48:00 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:48:00.709+0000 7ff459940700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.108705+0000 front 2024-02-21T23:47:24.108718+0000 (oldest deadline 2024-02-21T23:47:50.008543+0000) 2024-02-21T23:48:01.081 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:48:00 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:48:00.709+0000 7ff459940700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:30.009006+0000 front 2024-02-21T23:47:30.009050+0000 (oldest deadline 2024-02-21T23:47:57.009526+0000) 2024-02-21T23:48:01.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:00 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:00.895+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:48:01.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:00 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:00.895+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:48:01.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:00 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:00.895+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:48:01.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:00 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:00.895+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:38.458066+0000 front 2024-02-21T23:47:38.457888+0000 (oldest deadline 2024-02-21T23:48:00.757947+0000) 2024-02-21T23:48:01.336 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:01 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:01.093+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:48:01.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:01 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:01.093+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:48:01.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:01 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:01.093+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.160285+0000 front 2024-02-21T23:47:24.160271+0000 (oldest deadline 2024-02-21T23:47:48.860330+0000) 2024-02-21T23:48:01.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:01 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:01.093+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:32.361609+0000 front 2024-02-21T23:47:32.361527+0000 (oldest deadline 2024-02-21T23:47:54.661354+0000) 2024-02-21T23:48:01.481 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:01 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:01.178+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:48:01.481 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:01 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:01.178+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.583627+0000 front 2024-02-21T23:47:15.583731+0000 (oldest deadline 2024-02-21T23:47:40.883557+0000) 2024-02-21T23:48:01.482 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:01 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:01.178+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:21.384659+0000 front 2024-02-21T23:47:21.384611+0000 (oldest deadline 2024-02-21T23:47:47.284798+0000) 2024-02-21T23:48:01.482 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:01 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:01.178+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:30.785587+0000 front 2024-02-21T23:47:30.785598+0000 (oldest deadline 2024-02-21T23:47:55.485770+0000) 2024-02-21T23:48:01.678 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:01 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:01.410+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:05.990749+0000 front 2024-02-21T23:47:05.990685+0000 (oldest deadline 2024-02-21T23:47:32.991590+0000) 2024-02-21T23:48:01.679 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:01 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:01.410+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:18.193453+0000 front 2024-02-21T23:47:18.193604+0000 (oldest deadline 2024-02-21T23:47:41.693472+0000) 2024-02-21T23:48:01.679 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:01 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:01.410+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:23.394442+0000 front 2024-02-21T23:47:23.394464+0000 (oldest deadline 2024-02-21T23:47:46.894536+0000) 2024-02-21T23:48:01.679 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:01 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:01.410+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:31.595665+0000 front 2024-02-21T23:47:31.595563+0000 (oldest deadline 2024-02-21T23:47:57.495447+0000) 2024-02-21T23:48:01.679 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:01 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:01.410+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:38.497239+0000 front 2024-02-21T23:47:38.497020+0000 (oldest deadline 2024-02-21T23:48:00.796895+0000) 2024-02-21T23:48:02.087 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:48:01 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:48:01.694+0000 7ff459940700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.307063+0000 front 2024-02-21T23:47:15.307087+0000 (oldest deadline 2024-02-21T23:47:40.607300+0000) 2024-02-21T23:48:02.087 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:48:01 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:48:01.694+0000 7ff459940700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.108705+0000 front 2024-02-21T23:47:24.108718+0000 (oldest deadline 2024-02-21T23:47:50.008543+0000) 2024-02-21T23:48:02.088 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:48:01 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7[68762]: 2024-02-21T23:48:01.694+0000 7ff459940700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:30.009006+0000 front 2024-02-21T23:47:30.009050+0000 (oldest deadline 2024-02-21T23:47:57.009526+0000) 2024-02-21T23:48:02.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:01 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:01.885+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:48:02.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:01 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:01.885+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:48:02.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:01 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:01.885+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:48:02.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:01 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:01.885+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:38.458066+0000 front 2024-02-21T23:47:38.457888+0000 (oldest deadline 2024-02-21T23:48:00.757947+0000) 2024-02-21T23:48:02.481 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:02 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:02.177+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:48:02.481 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:02 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:02.177+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.583627+0000 front 2024-02-21T23:47:15.583731+0000 (oldest deadline 2024-02-21T23:47:40.883557+0000) 2024-02-21T23:48:02.481 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:02 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:02.177+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:21.384659+0000 front 2024-02-21T23:47:21.384611+0000 (oldest deadline 2024-02-21T23:47:47.284798+0000) 2024-02-21T23:48:02.481 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:02 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:02.177+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:30.785587+0000 front 2024-02-21T23:47:30.785598+0000 (oldest deadline 2024-02-21T23:47:55.485770+0000) 2024-02-21T23:48:02.587 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 21 23:48:02 smithi176 bash[85804]: ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-7 2024-02-21T23:48:02.588 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:02 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:02.134+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:48:02.588 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:02 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:02.134+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:48:02.588 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:02 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:02.134+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.160285+0000 front 2024-02-21T23:47:24.160271+0000 (oldest deadline 2024-02-21T23:47:48.860330+0000) 2024-02-21T23:48:02.588 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:02 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:02.134+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:32.361609+0000 front 2024-02-21T23:47:32.361527+0000 (oldest deadline 2024-02-21T23:47:54.661354+0000) 2024-02-21T23:48:02.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:02 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:02.419+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:05.990749+0000 front 2024-02-21T23:47:05.990685+0000 (oldest deadline 2024-02-21T23:47:32.991590+0000) 2024-02-21T23:48:02.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:02 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:02.419+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:18.193453+0000 front 2024-02-21T23:47:18.193604+0000 (oldest deadline 2024-02-21T23:47:41.693472+0000) 2024-02-21T23:48:02.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:02 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:02.419+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:23.394442+0000 front 2024-02-21T23:47:23.394464+0000 (oldest deadline 2024-02-21T23:47:46.894536+0000) 2024-02-21T23:48:02.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:02 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:02.419+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:31.595665+0000 front 2024-02-21T23:47:31.595563+0000 (oldest deadline 2024-02-21T23:47:57.495447+0000) 2024-02-21T23:48:02.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:02 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:02.419+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:38.497239+0000 front 2024-02-21T23:47:38.497020+0000 (oldest deadline 2024-02-21T23:48:00.796895+0000) 2024-02-21T23:48:03.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:02 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:02.907+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:48:03.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:02 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:02.907+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:48:03.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:02 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:02.907+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:48:03.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:02 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:02.907+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:38.458066+0000 front 2024-02-21T23:47:38.457888+0000 (oldest deadline 2024-02-21T23:48:00.757947+0000) 2024-02-21T23:48:03.414 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:03 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:03.160+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:48:03.414 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:03 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:03.160+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:48:03.414 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:03 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:03.160+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.160285+0000 front 2024-02-21T23:47:24.160271+0000 (oldest deadline 2024-02-21T23:47:48.860330+0000) 2024-02-21T23:48:03.414 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:03 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:03.160+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:32.361609+0000 front 2024-02-21T23:47:32.361527+0000 (oldest deadline 2024-02-21T23:47:54.661354+0000) 2024-02-21T23:48:03.420 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@osd.7.service' 2024-02-21T23:48:03.462 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-21T23:48:03.462 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-02-21T23:48:03.463 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-02-21T23:48:03.463 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-7ed81218-d112-11ee-95bd-87774f69a715@osd.8 2024-02-21T23:48:03.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:03 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:03.184+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:48:03.481 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:03 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:03.184+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.583627+0000 front 2024-02-21T23:47:15.583731+0000 (oldest deadline 2024-02-21T23:47:40.883557+0000) 2024-02-21T23:48:03.481 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:03 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:03.184+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:21.384659+0000 front 2024-02-21T23:47:21.384611+0000 (oldest deadline 2024-02-21T23:47:47.284798+0000) 2024-02-21T23:48:03.481 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:03 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:03.184+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:30.785587+0000 front 2024-02-21T23:47:30.785598+0000 (oldest deadline 2024-02-21T23:47:55.485770+0000) 2024-02-21T23:48:03.481 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:03 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:03.184+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:40.087567+0000 front 2024-02-21T23:47:40.087558+0000 (oldest deadline 2024-02-21T23:48:02.887757+0000) 2024-02-21T23:48:03.724 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:03 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:03.459+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:05.990749+0000 front 2024-02-21T23:47:05.990685+0000 (oldest deadline 2024-02-21T23:47:32.991590+0000) 2024-02-21T23:48:03.724 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:03 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:03.459+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:18.193453+0000 front 2024-02-21T23:47:18.193604+0000 (oldest deadline 2024-02-21T23:47:41.693472+0000) 2024-02-21T23:48:03.724 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:03 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:03.459+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:23.394442+0000 front 2024-02-21T23:47:23.394464+0000 (oldest deadline 2024-02-21T23:47:46.894536+0000) 2024-02-21T23:48:03.724 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:03 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:03.459+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:31.595665+0000 front 2024-02-21T23:47:31.595563+0000 (oldest deadline 2024-02-21T23:47:57.495447+0000) 2024-02-21T23:48:03.724 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:03 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:03.459+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:38.497239+0000 front 2024-02-21T23:47:38.497020+0000 (oldest deadline 2024-02-21T23:48:00.796895+0000) 2024-02-21T23:48:03.725 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:03 smithi178 systemd[1]: Stopping Ceph osd.8 for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:48:03.725 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:03 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:03.736+0000 7ff479023700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-21T23:48:03.725 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:03 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:03.736+0000 7ff479023700 -1 osd.8 85 *** Got signal Terminated *** 2024-02-21T23:48:04.177 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:03 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:03.736+0000 7ff479023700 -1 osd.8 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-21T23:48:04.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:03 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:03.938+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:48:04.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:03 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:03.938+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:48:04.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:03 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:03.938+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:48:04.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:03 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:03.938+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:38.458066+0000 front 2024-02-21T23:47:38.457888+0000 (oldest deadline 2024-02-21T23:48:00.757947+0000) 2024-02-21T23:48:04.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:04 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:04.207+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:48:04.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:04 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:04.207+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.583627+0000 front 2024-02-21T23:47:15.583731+0000 (oldest deadline 2024-02-21T23:47:40.883557+0000) 2024-02-21T23:48:04.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:04 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:04.207+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:21.384659+0000 front 2024-02-21T23:47:21.384611+0000 (oldest deadline 2024-02-21T23:47:47.284798+0000) 2024-02-21T23:48:04.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:04 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:04.207+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:30.785587+0000 front 2024-02-21T23:47:30.785598+0000 (oldest deadline 2024-02-21T23:47:55.485770+0000) 2024-02-21T23:48:04.480 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:04 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:04.207+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:40.087567+0000 front 2024-02-21T23:47:40.087558+0000 (oldest deadline 2024-02-21T23:48:02.887757+0000) 2024-02-21T23:48:04.586 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:04 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:04.173+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:48:04.587 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:04 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:04.173+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:48:04.587 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:04 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:04.173+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.160285+0000 front 2024-02-21T23:47:24.160271+0000 (oldest deadline 2024-02-21T23:47:48.860330+0000) 2024-02-21T23:48:04.587 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:04 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:04.173+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:32.361609+0000 front 2024-02-21T23:47:32.361527+0000 (oldest deadline 2024-02-21T23:47:54.661354+0000) 2024-02-21T23:48:04.897 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:04 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:04.501+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:05.990749+0000 front 2024-02-21T23:47:05.990685+0000 (oldest deadline 2024-02-21T23:47:32.991590+0000) 2024-02-21T23:48:04.897 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:04 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:04.501+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:18.193453+0000 front 2024-02-21T23:47:18.193604+0000 (oldest deadline 2024-02-21T23:47:41.693472+0000) 2024-02-21T23:48:04.897 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:04 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:04.501+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:23.394442+0000 front 2024-02-21T23:47:23.394464+0000 (oldest deadline 2024-02-21T23:47:46.894536+0000) 2024-02-21T23:48:04.897 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:04 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:04.501+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:31.595665+0000 front 2024-02-21T23:47:31.595563+0000 (oldest deadline 2024-02-21T23:47:57.495447+0000) 2024-02-21T23:48:04.897 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:04 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:04.501+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:38.497239+0000 front 2024-02-21T23:47:38.497020+0000 (oldest deadline 2024-02-21T23:48:00.796895+0000) 2024-02-21T23:48:05.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:04 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:04.909+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:48:05.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:04 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:04.909+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:48:05.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:04 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:04.909+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:48:05.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:04 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:04.909+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:38.458066+0000 front 2024-02-21T23:47:38.457888+0000 (oldest deadline 2024-02-21T23:48:00.757947+0000) 2024-02-21T23:48:05.587 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:05 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:05.124+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:48:05.587 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:05 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:05.124+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:48:05.587 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:05 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:05.124+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.160285+0000 front 2024-02-21T23:47:24.160271+0000 (oldest deadline 2024-02-21T23:47:48.860330+0000) 2024-02-21T23:48:05.587 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:05 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:05.124+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:32.361609+0000 front 2024-02-21T23:47:32.361527+0000 (oldest deadline 2024-02-21T23:47:54.661354+0000) 2024-02-21T23:48:05.587 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:05 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:05.124+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:39.962203+0000 front 2024-02-21T23:47:39.962151+0000 (oldest deadline 2024-02-21T23:48:04.662264+0000) 2024-02-21T23:48:05.730 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:05 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:05.245+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:48:05.730 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:05 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:05.245+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.583627+0000 front 2024-02-21T23:47:15.583731+0000 (oldest deadline 2024-02-21T23:47:40.883557+0000) 2024-02-21T23:48:05.730 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:05 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:05.245+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:21.384659+0000 front 2024-02-21T23:47:21.384611+0000 (oldest deadline 2024-02-21T23:47:47.284798+0000) 2024-02-21T23:48:05.730 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:05 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:05.246+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:30.785587+0000 front 2024-02-21T23:47:30.785598+0000 (oldest deadline 2024-02-21T23:47:55.485770+0000) 2024-02-21T23:48:05.730 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:05 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:05.246+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:40.087567+0000 front 2024-02-21T23:47:40.087558+0000 (oldest deadline 2024-02-21T23:48:02.887757+0000) 2024-02-21T23:48:05.869 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:05 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:05.467+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:05.990749+0000 front 2024-02-21T23:47:05.990685+0000 (oldest deadline 2024-02-21T23:47:32.991590+0000) 2024-02-21T23:48:05.869 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:05 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:05.467+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:18.193453+0000 front 2024-02-21T23:47:18.193604+0000 (oldest deadline 2024-02-21T23:47:41.693472+0000) 2024-02-21T23:48:05.869 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:05 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:05.467+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:23.394442+0000 front 2024-02-21T23:47:23.394464+0000 (oldest deadline 2024-02-21T23:47:46.894536+0000) 2024-02-21T23:48:05.869 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:05 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:05.467+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:31.595665+0000 front 2024-02-21T23:47:31.595563+0000 (oldest deadline 2024-02-21T23:47:57.495447+0000) 2024-02-21T23:48:05.870 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:05 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:05.467+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:38.497239+0000 front 2024-02-21T23:47:38.497020+0000 (oldest deadline 2024-02-21T23:48:00.796895+0000) 2024-02-21T23:48:06.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:05 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:05.882+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:48:06.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:05 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:05.882+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:48:06.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:05 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:05.882+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:48:06.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:05 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:05.882+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:38.458066+0000 front 2024-02-21T23:47:38.457888+0000 (oldest deadline 2024-02-21T23:48:00.757947+0000) 2024-02-21T23:48:06.587 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:06 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:06.107+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:48:06.587 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:06 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:06.107+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:48:06.587 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:06 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:06.107+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.160285+0000 front 2024-02-21T23:47:24.160271+0000 (oldest deadline 2024-02-21T23:47:48.860330+0000) 2024-02-21T23:48:06.587 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:06 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:06.107+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:32.361609+0000 front 2024-02-21T23:47:32.361527+0000 (oldest deadline 2024-02-21T23:47:54.661354+0000) 2024-02-21T23:48:06.587 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:06 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:06.107+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:39.962203+0000 front 2024-02-21T23:47:39.962151+0000 (oldest deadline 2024-02-21T23:48:04.662264+0000) 2024-02-21T23:48:06.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:06 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:06.439+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:05.990749+0000 front 2024-02-21T23:47:05.990685+0000 (oldest deadline 2024-02-21T23:47:32.991590+0000) 2024-02-21T23:48:06.678 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:06 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:06.439+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:18.193453+0000 front 2024-02-21T23:47:18.193604+0000 (oldest deadline 2024-02-21T23:47:41.693472+0000) 2024-02-21T23:48:06.678 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:06 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:06.439+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:23.394442+0000 front 2024-02-21T23:47:23.394464+0000 (oldest deadline 2024-02-21T23:47:46.894536+0000) 2024-02-21T23:48:06.678 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:06 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:06.439+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:31.595665+0000 front 2024-02-21T23:47:31.595563+0000 (oldest deadline 2024-02-21T23:47:57.495447+0000) 2024-02-21T23:48:06.678 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:06 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:06.439+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:38.497239+0000 front 2024-02-21T23:47:38.497020+0000 (oldest deadline 2024-02-21T23:48:00.796895+0000) 2024-02-21T23:48:06.730 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:06 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:06.279+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:48:06.730 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:06 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:06.279+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.583627+0000 front 2024-02-21T23:47:15.583731+0000 (oldest deadline 2024-02-21T23:47:40.883557+0000) 2024-02-21T23:48:06.730 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:06 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:06.279+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:21.384659+0000 front 2024-02-21T23:47:21.384611+0000 (oldest deadline 2024-02-21T23:47:47.284798+0000) 2024-02-21T23:48:06.730 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:06 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:06.279+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:30.785587+0000 front 2024-02-21T23:47:30.785598+0000 (oldest deadline 2024-02-21T23:47:55.485770+0000) 2024-02-21T23:48:06.730 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:06 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:06.279+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:40.087567+0000 front 2024-02-21T23:47:40.087558+0000 (oldest deadline 2024-02-21T23:48:02.887757+0000) 2024-02-21T23:48:07.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:06 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:06.886+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:48:07.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:06 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:06.886+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:48:07.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:06 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:06.886+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:48:07.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:06 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:06.886+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:38.458066+0000 front 2024-02-21T23:47:38.457888+0000 (oldest deadline 2024-02-21T23:48:00.757947+0000) 2024-02-21T23:48:07.587 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:07 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:07.113+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:48:07.587 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:07 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:07.114+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:48:07.587 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:07 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:07.114+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.160285+0000 front 2024-02-21T23:47:24.160271+0000 (oldest deadline 2024-02-21T23:47:48.860330+0000) 2024-02-21T23:48:07.587 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:07 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:07.114+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:32.361609+0000 front 2024-02-21T23:47:32.361527+0000 (oldest deadline 2024-02-21T23:47:54.661354+0000) 2024-02-21T23:48:07.587 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:07 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:07.114+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:39.962203+0000 front 2024-02-21T23:47:39.962151+0000 (oldest deadline 2024-02-21T23:48:04.662264+0000) 2024-02-21T23:48:07.730 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:07 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:07.275+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:48:07.730 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:07 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:07.275+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.583627+0000 front 2024-02-21T23:47:15.583731+0000 (oldest deadline 2024-02-21T23:47:40.883557+0000) 2024-02-21T23:48:07.730 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:07 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:07.275+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:21.384659+0000 front 2024-02-21T23:47:21.384611+0000 (oldest deadline 2024-02-21T23:47:47.284798+0000) 2024-02-21T23:48:07.731 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:07 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:07.275+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:30.785587+0000 front 2024-02-21T23:47:30.785598+0000 (oldest deadline 2024-02-21T23:47:55.485770+0000) 2024-02-21T23:48:07.731 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:07 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:07.275+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:40.087567+0000 front 2024-02-21T23:47:40.087558+0000 (oldest deadline 2024-02-21T23:48:02.887757+0000) 2024-02-21T23:48:07.835 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:07 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:07.488+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:05.990749+0000 front 2024-02-21T23:47:05.990685+0000 (oldest deadline 2024-02-21T23:47:32.991590+0000) 2024-02-21T23:48:07.835 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:07 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:07.488+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:18.193453+0000 front 2024-02-21T23:47:18.193604+0000 (oldest deadline 2024-02-21T23:47:41.693472+0000) 2024-02-21T23:48:07.835 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:07 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:07.488+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:23.394442+0000 front 2024-02-21T23:47:23.394464+0000 (oldest deadline 2024-02-21T23:47:46.894536+0000) 2024-02-21T23:48:07.835 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:07 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:07.488+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:31.595665+0000 front 2024-02-21T23:47:31.595563+0000 (oldest deadline 2024-02-21T23:47:57.495447+0000) 2024-02-21T23:48:07.835 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:07 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:07.488+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:38.497239+0000 front 2024-02-21T23:47:38.497020+0000 (oldest deadline 2024-02-21T23:48:00.796895+0000) 2024-02-21T23:48:08.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:07 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:07.848+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:48:08.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:07 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:07.848+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:48:08.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:07 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:07.848+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:48:08.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:07 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:07.848+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:38.458066+0000 front 2024-02-21T23:47:38.457888+0000 (oldest deadline 2024-02-21T23:48:00.757947+0000) 2024-02-21T23:48:08.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:08 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:08.069+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:48:08.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:08 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:08.069+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:48:08.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:08 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:08.069+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.160285+0000 front 2024-02-21T23:47:24.160271+0000 (oldest deadline 2024-02-21T23:47:48.860330+0000) 2024-02-21T23:48:08.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:08 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:08.069+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:32.361609+0000 front 2024-02-21T23:47:32.361527+0000 (oldest deadline 2024-02-21T23:47:54.661354+0000) 2024-02-21T23:48:08.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:08 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:08.069+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:39.962203+0000 front 2024-02-21T23:47:39.962151+0000 (oldest deadline 2024-02-21T23:48:04.662264+0000) 2024-02-21T23:48:08.730 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:08 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:08.247+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:48:08.730 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:08 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:08.247+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.583627+0000 front 2024-02-21T23:47:15.583731+0000 (oldest deadline 2024-02-21T23:47:40.883557+0000) 2024-02-21T23:48:08.730 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:08 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:08.247+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:21.384659+0000 front 2024-02-21T23:47:21.384611+0000 (oldest deadline 2024-02-21T23:47:47.284798+0000) 2024-02-21T23:48:08.730 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:08 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:08.247+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:30.785587+0000 front 2024-02-21T23:47:30.785598+0000 (oldest deadline 2024-02-21T23:47:55.485770+0000) 2024-02-21T23:48:08.731 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:08 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:08.247+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:40.087567+0000 front 2024-02-21T23:47:40.087558+0000 (oldest deadline 2024-02-21T23:48:02.887757+0000) 2024-02-21T23:48:08.797 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:08 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:08.445+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:05.990749+0000 front 2024-02-21T23:47:05.990685+0000 (oldest deadline 2024-02-21T23:47:32.991590+0000) 2024-02-21T23:48:08.797 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:08 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:08.445+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:18.193453+0000 front 2024-02-21T23:47:18.193604+0000 (oldest deadline 2024-02-21T23:47:41.693472+0000) 2024-02-21T23:48:08.797 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:08 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:08.445+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:23.394442+0000 front 2024-02-21T23:47:23.394464+0000 (oldest deadline 2024-02-21T23:47:46.894536+0000) 2024-02-21T23:48:08.797 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:08 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:08.445+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:31.595665+0000 front 2024-02-21T23:47:31.595563+0000 (oldest deadline 2024-02-21T23:47:57.495447+0000) 2024-02-21T23:48:08.797 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:08 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8[70539]: 2024-02-21T23:48:08.445+0000 7ff475619700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:38.497239+0000 front 2024-02-21T23:47:38.497020+0000 (oldest deadline 2024-02-21T23:48:00.796895+0000) 2024-02-21T23:48:09.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:08 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:08.809+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:48:09.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:08 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:08.809+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:48:09.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:08 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:08.809+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:48:09.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:08 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:08.809+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:38.458066+0000 front 2024-02-21T23:47:38.457888+0000 (oldest deadline 2024-02-21T23:48:00.757947+0000) 2024-02-21T23:48:09.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:08 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:08.809+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:46.559120+0000 front 2024-02-21T23:47:46.559318+0000 (oldest deadline 2024-02-21T23:48:08.259195+0000) 2024-02-21T23:48:09.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:09 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:09.037+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:48:09.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:09 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:09.037+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:48:09.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:09 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:09.037+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.160285+0000 front 2024-02-21T23:47:24.160271+0000 (oldest deadline 2024-02-21T23:47:48.860330+0000) 2024-02-21T23:48:09.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:09 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:09.037+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:32.361609+0000 front 2024-02-21T23:47:32.361527+0000 (oldest deadline 2024-02-21T23:47:54.661354+0000) 2024-02-21T23:48:09.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:09 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:09.037+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:39.962203+0000 front 2024-02-21T23:47:39.962151+0000 (oldest deadline 2024-02-21T23:48:04.662264+0000) 2024-02-21T23:48:09.677 INFO:journalctl@ceph.osd.8.smithi178.stdout:Feb 21 23:48:09 smithi178 bash[87899]: ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-8 2024-02-21T23:48:09.730 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:09 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:09.265+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:48:09.730 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:09 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:09.265+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.583627+0000 front 2024-02-21T23:47:15.583731+0000 (oldest deadline 2024-02-21T23:47:40.883557+0000) 2024-02-21T23:48:09.730 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:09 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:09.265+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:21.384659+0000 front 2024-02-21T23:47:21.384611+0000 (oldest deadline 2024-02-21T23:47:47.284798+0000) 2024-02-21T23:48:09.730 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:09 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:09.265+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:30.785587+0000 front 2024-02-21T23:47:30.785598+0000 (oldest deadline 2024-02-21T23:47:55.485770+0000) 2024-02-21T23:48:09.731 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:09 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:09.265+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:40.087567+0000 front 2024-02-21T23:47:40.087558+0000 (oldest deadline 2024-02-21T23:48:02.887757+0000) 2024-02-21T23:48:09.731 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:09 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:09.265+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:42.888385+0000 front 2024-02-21T23:47:42.888319+0000 (oldest deadline 2024-02-21T23:48:08.788343+0000) 2024-02-21T23:48:10.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:09 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:09.812+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:48:10.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:09 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:09.813+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:48:10.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:09 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:09.813+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:48:10.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:09 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:09.813+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:38.458066+0000 front 2024-02-21T23:47:38.457888+0000 (oldest deadline 2024-02-21T23:48:00.757947+0000) 2024-02-21T23:48:10.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:09 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:09.813+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:46.559120+0000 front 2024-02-21T23:47:46.559318+0000 (oldest deadline 2024-02-21T23:48:08.259195+0000) 2024-02-21T23:48:10.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:10 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:10.034+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:48:10.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:10 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:10.034+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:48:10.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:10 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:10.034+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.160285+0000 front 2024-02-21T23:47:24.160271+0000 (oldest deadline 2024-02-21T23:47:48.860330+0000) 2024-02-21T23:48:10.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:10 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:10.034+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:32.361609+0000 front 2024-02-21T23:47:32.361527+0000 (oldest deadline 2024-02-21T23:47:54.661354+0000) 2024-02-21T23:48:10.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:10 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:10.034+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:39.962203+0000 front 2024-02-21T23:47:39.962151+0000 (oldest deadline 2024-02-21T23:48:04.662264+0000) 2024-02-21T23:48:10.602 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@osd.8.service' 2024-02-21T23:48:10.642 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-21T23:48:10.642 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-02-21T23:48:10.642 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-02-21T23:48:10.642 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-7ed81218-d112-11ee-95bd-87774f69a715@osd.9 2024-02-21T23:48:10.672 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:10 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:10.269+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:48:10.672 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:10 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:10.269+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.583627+0000 front 2024-02-21T23:47:15.583731+0000 (oldest deadline 2024-02-21T23:47:40.883557+0000) 2024-02-21T23:48:10.672 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:10 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:10.269+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:21.384659+0000 front 2024-02-21T23:47:21.384611+0000 (oldest deadline 2024-02-21T23:47:47.284798+0000) 2024-02-21T23:48:10.672 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:10 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:10.269+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:30.785587+0000 front 2024-02-21T23:47:30.785598+0000 (oldest deadline 2024-02-21T23:47:55.485770+0000) 2024-02-21T23:48:10.672 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:10 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:10.269+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:40.087567+0000 front 2024-02-21T23:47:40.087558+0000 (oldest deadline 2024-02-21T23:48:02.887757+0000) 2024-02-21T23:48:10.672 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:10 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:10.269+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:42.888385+0000 front 2024-02-21T23:47:42.888319+0000 (oldest deadline 2024-02-21T23:48:08.788343+0000) 2024-02-21T23:48:10.927 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:10 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:10.769+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:48:10.927 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:10 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:10.769+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:48:10.927 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:10 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:10.769+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:48:10.927 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:10 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:10.769+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:38.458066+0000 front 2024-02-21T23:47:38.457888+0000 (oldest deadline 2024-02-21T23:48:00.757947+0000) 2024-02-21T23:48:10.927 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:10 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:10.769+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:46.559120+0000 front 2024-02-21T23:47:46.559318+0000 (oldest deadline 2024-02-21T23:48:08.259195+0000) 2024-02-21T23:48:10.980 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:10 smithi090 systemd[1]: Stopping Ceph osd.9 for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:48:10.980 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:10 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:10.894+0000 7f35ece33700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-21T23:48:10.980 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:10 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:10.894+0000 7f35ece33700 -1 osd.9 85 *** Got signal Terminated *** 2024-02-21T23:48:10.980 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:10 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:10.894+0000 7f35ece33700 -1 osd.9 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-21T23:48:11.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:11 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:11.013+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:48:11.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:11 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:11.013+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:48:11.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:11 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:11.013+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.160285+0000 front 2024-02-21T23:47:24.160271+0000 (oldest deadline 2024-02-21T23:47:48.860330+0000) 2024-02-21T23:48:11.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:11 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:11.013+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:32.361609+0000 front 2024-02-21T23:47:32.361527+0000 (oldest deadline 2024-02-21T23:47:54.661354+0000) 2024-02-21T23:48:11.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:11 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:11.013+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:39.962203+0000 front 2024-02-21T23:47:39.962151+0000 (oldest deadline 2024-02-21T23:48:04.662264+0000) 2024-02-21T23:48:11.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:11 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:11.013+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:45.163315+0000 front 2024-02-21T23:47:45.163402+0000 (oldest deadline 2024-02-21T23:48:10.463241+0000) 2024-02-21T23:48:11.730 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:11 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:11.279+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:48:11.730 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:11 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:11.279+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.583627+0000 front 2024-02-21T23:47:15.583731+0000 (oldest deadline 2024-02-21T23:47:40.883557+0000) 2024-02-21T23:48:11.730 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:11 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:11.279+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:21.384659+0000 front 2024-02-21T23:47:21.384611+0000 (oldest deadline 2024-02-21T23:47:47.284798+0000) 2024-02-21T23:48:11.731 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:11 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:11.279+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:30.785587+0000 front 2024-02-21T23:47:30.785598+0000 (oldest deadline 2024-02-21T23:47:55.485770+0000) 2024-02-21T23:48:11.731 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:11 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:11.279+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:40.087567+0000 front 2024-02-21T23:47:40.087558+0000 (oldest deadline 2024-02-21T23:48:02.887757+0000) 2024-02-21T23:48:11.731 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:11 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:11.279+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:42.888385+0000 front 2024-02-21T23:47:42.888319+0000 (oldest deadline 2024-02-21T23:48:08.788343+0000) 2024-02-21T23:48:12.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:11 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:11.818+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:48:12.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:11 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:11.818+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:48:12.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:11 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:11.818+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:48:12.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:11 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:11.818+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:38.458066+0000 front 2024-02-21T23:47:38.457888+0000 (oldest deadline 2024-02-21T23:48:00.757947+0000) 2024-02-21T23:48:12.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:11 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:11.818+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:46.559120+0000 front 2024-02-21T23:47:46.559318+0000 (oldest deadline 2024-02-21T23:48:08.259195+0000) 2024-02-21T23:48:12.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:12 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:12.053+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:48:12.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:12 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:12.053+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:48:12.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:12 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:12.053+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.160285+0000 front 2024-02-21T23:47:24.160271+0000 (oldest deadline 2024-02-21T23:47:48.860330+0000) 2024-02-21T23:48:12.338 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:12 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:12.053+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:32.361609+0000 front 2024-02-21T23:47:32.361527+0000 (oldest deadline 2024-02-21T23:47:54.661354+0000) 2024-02-21T23:48:12.338 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:12 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:12.053+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:39.962203+0000 front 2024-02-21T23:47:39.962151+0000 (oldest deadline 2024-02-21T23:48:04.662264+0000) 2024-02-21T23:48:12.338 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:12 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:12.053+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:45.163315+0000 front 2024-02-21T23:47:45.163402+0000 (oldest deadline 2024-02-21T23:48:10.463241+0000) 2024-02-21T23:48:12.730 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:12 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:12.321+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:48:12.731 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:12 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:12.321+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.583627+0000 front 2024-02-21T23:47:15.583731+0000 (oldest deadline 2024-02-21T23:47:40.883557+0000) 2024-02-21T23:48:12.731 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:12 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:12.321+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:21.384659+0000 front 2024-02-21T23:47:21.384611+0000 (oldest deadline 2024-02-21T23:47:47.284798+0000) 2024-02-21T23:48:12.731 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:12 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:12.321+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:30.785587+0000 front 2024-02-21T23:47:30.785598+0000 (oldest deadline 2024-02-21T23:47:55.485770+0000) 2024-02-21T23:48:12.731 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:12 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:12.321+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:40.087567+0000 front 2024-02-21T23:47:40.087558+0000 (oldest deadline 2024-02-21T23:48:02.887757+0000) 2024-02-21T23:48:12.731 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:12 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:12.321+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:42.888385+0000 front 2024-02-21T23:47:42.888319+0000 (oldest deadline 2024-02-21T23:48:08.788343+0000) 2024-02-21T23:48:13.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:12 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:12.848+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:48:13.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:12 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:12.848+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:48:13.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:12 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:12.848+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:48:13.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:12 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:12.848+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:38.458066+0000 front 2024-02-21T23:47:38.457888+0000 (oldest deadline 2024-02-21T23:48:00.757947+0000) 2024-02-21T23:48:13.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:12 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:12.848+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:46.559120+0000 front 2024-02-21T23:47:46.559318+0000 (oldest deadline 2024-02-21T23:48:08.259195+0000) 2024-02-21T23:48:13.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:13 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:13.036+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:48:13.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:13 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:13.036+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:48:13.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:13 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:13.036+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.160285+0000 front 2024-02-21T23:47:24.160271+0000 (oldest deadline 2024-02-21T23:47:48.860330+0000) 2024-02-21T23:48:13.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:13 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:13.036+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:32.361609+0000 front 2024-02-21T23:47:32.361527+0000 (oldest deadline 2024-02-21T23:47:54.661354+0000) 2024-02-21T23:48:13.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:13 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:13.036+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:39.962203+0000 front 2024-02-21T23:47:39.962151+0000 (oldest deadline 2024-02-21T23:48:04.662264+0000) 2024-02-21T23:48:13.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:13 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:13.036+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:45.163315+0000 front 2024-02-21T23:47:45.163402+0000 (oldest deadline 2024-02-21T23:48:10.463241+0000) 2024-02-21T23:48:13.730 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:13 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:13.321+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:48:13.731 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:13 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:13.321+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.583627+0000 front 2024-02-21T23:47:15.583731+0000 (oldest deadline 2024-02-21T23:47:40.883557+0000) 2024-02-21T23:48:13.731 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:13 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:13.321+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:21.384659+0000 front 2024-02-21T23:47:21.384611+0000 (oldest deadline 2024-02-21T23:47:47.284798+0000) 2024-02-21T23:48:13.731 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:13 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:13.321+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:30.785587+0000 front 2024-02-21T23:47:30.785598+0000 (oldest deadline 2024-02-21T23:47:55.485770+0000) 2024-02-21T23:48:13.731 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:13 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:13.321+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:40.087567+0000 front 2024-02-21T23:47:40.087558+0000 (oldest deadline 2024-02-21T23:48:02.887757+0000) 2024-02-21T23:48:13.731 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:13 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:13.321+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:42.888385+0000 front 2024-02-21T23:47:42.888319+0000 (oldest deadline 2024-02-21T23:48:08.788343+0000) 2024-02-21T23:48:14.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:13 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:13.857+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:48:14.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:13 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:13.857+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:48:14.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:13 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:13.857+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:48:14.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:13 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:13.857+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:38.458066+0000 front 2024-02-21T23:47:38.457888+0000 (oldest deadline 2024-02-21T23:48:00.757947+0000) 2024-02-21T23:48:14.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:13 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:13.857+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:46.559120+0000 front 2024-02-21T23:47:46.559318+0000 (oldest deadline 2024-02-21T23:48:08.259195+0000) 2024-02-21T23:48:14.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:14 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:14.019+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:48:14.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:14 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:14.019+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:48:14.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:14 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:14.019+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.160285+0000 front 2024-02-21T23:47:24.160271+0000 (oldest deadline 2024-02-21T23:47:48.860330+0000) 2024-02-21T23:48:14.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:14 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:14.019+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:32.361609+0000 front 2024-02-21T23:47:32.361527+0000 (oldest deadline 2024-02-21T23:47:54.661354+0000) 2024-02-21T23:48:14.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:14 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:14.019+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:39.962203+0000 front 2024-02-21T23:47:39.962151+0000 (oldest deadline 2024-02-21T23:48:04.662264+0000) 2024-02-21T23:48:14.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:14 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:14.019+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:45.163315+0000 front 2024-02-21T23:47:45.163402+0000 (oldest deadline 2024-02-21T23:48:10.463241+0000) 2024-02-21T23:48:14.730 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:14 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:14.313+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:48:14.731 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:14 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:14.313+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.583627+0000 front 2024-02-21T23:47:15.583731+0000 (oldest deadline 2024-02-21T23:47:40.883557+0000) 2024-02-21T23:48:14.731 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:14 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:14.313+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:21.384659+0000 front 2024-02-21T23:47:21.384611+0000 (oldest deadline 2024-02-21T23:47:47.284798+0000) 2024-02-21T23:48:14.731 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:14 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:14.313+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:30.785587+0000 front 2024-02-21T23:47:30.785598+0000 (oldest deadline 2024-02-21T23:47:55.485770+0000) 2024-02-21T23:48:14.731 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:14 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:14.313+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:40.087567+0000 front 2024-02-21T23:47:40.087558+0000 (oldest deadline 2024-02-21T23:48:02.887757+0000) 2024-02-21T23:48:14.731 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:14 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:14.313+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:42.888385+0000 front 2024-02-21T23:47:42.888319+0000 (oldest deadline 2024-02-21T23:48:08.788343+0000) 2024-02-21T23:48:15.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:14 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:14.872+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:48:15.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:14 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:14.872+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:48:15.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:14 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:14.872+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:48:15.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:14 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:14.872+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:38.458066+0000 front 2024-02-21T23:47:38.457888+0000 (oldest deadline 2024-02-21T23:48:00.757947+0000) 2024-02-21T23:48:15.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:14 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:14.872+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:46.559120+0000 front 2024-02-21T23:47:46.559318+0000 (oldest deadline 2024-02-21T23:48:08.259195+0000) 2024-02-21T23:48:15.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:14 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:14.973+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:48:15.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:14 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:14.973+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:48:15.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:14 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:14.973+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.160285+0000 front 2024-02-21T23:47:24.160271+0000 (oldest deadline 2024-02-21T23:47:48.860330+0000) 2024-02-21T23:48:15.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:14 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:14.973+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:32.361609+0000 front 2024-02-21T23:47:32.361527+0000 (oldest deadline 2024-02-21T23:47:54.661354+0000) 2024-02-21T23:48:15.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:14 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:14.973+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:39.962203+0000 front 2024-02-21T23:47:39.962151+0000 (oldest deadline 2024-02-21T23:48:04.662264+0000) 2024-02-21T23:48:15.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:14 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:14.973+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:45.163315+0000 front 2024-02-21T23:47:45.163402+0000 (oldest deadline 2024-02-21T23:48:10.463241+0000) 2024-02-21T23:48:15.338 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:14 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:14.973+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-02-21T23:47:53.865163+0000 front 2024-02-21T23:47:53.865128+0000 (oldest deadline 2024-02-21T23:48:14.965109+0000) 2024-02-21T23:48:15.730 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:15 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:15.357+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:07.982227+0000 front 2024-02-21T23:47:07.982265+0000 (oldest deadline 2024-02-21T23:47:32.682449+0000) 2024-02-21T23:48:15.730 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:15 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:15.357+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:15.583627+0000 front 2024-02-21T23:47:15.583731+0000 (oldest deadline 2024-02-21T23:47:40.883557+0000) 2024-02-21T23:48:15.731 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:15 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:15.357+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:21.384659+0000 front 2024-02-21T23:47:21.384611+0000 (oldest deadline 2024-02-21T23:47:47.284798+0000) 2024-02-21T23:48:15.731 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:15 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:15.357+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:30.785587+0000 front 2024-02-21T23:47:30.785598+0000 (oldest deadline 2024-02-21T23:47:55.485770+0000) 2024-02-21T23:48:15.731 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:15 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:15.357+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:40.087567+0000 front 2024-02-21T23:47:40.087558+0000 (oldest deadline 2024-02-21T23:48:02.887757+0000) 2024-02-21T23:48:15.731 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:15 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9[86651]: 2024-02-21T23:48:15.357+0000 7f35e7425700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:42.888385+0000 front 2024-02-21T23:47:42.888319+0000 (oldest deadline 2024-02-21T23:48:08.788343+0000) 2024-02-21T23:48:16.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:15 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:15.882+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:48:16.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:15 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:15.882+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:48:16.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:15 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:15.882+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:48:16.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:15 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:15.882+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:38.458066+0000 front 2024-02-21T23:47:38.457888+0000 (oldest deadline 2024-02-21T23:48:00.757947+0000) 2024-02-21T23:48:16.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:15 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:15.882+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:46.559120+0000 front 2024-02-21T23:47:46.559318+0000 (oldest deadline 2024-02-21T23:48:08.259195+0000) 2024-02-21T23:48:16.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:15 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:15.953+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:48:16.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:15 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:15.953+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:48:16.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:15 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:15.953+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.160285+0000 front 2024-02-21T23:47:24.160271+0000 (oldest deadline 2024-02-21T23:47:48.860330+0000) 2024-02-21T23:48:16.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:15 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:15.953+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:32.361609+0000 front 2024-02-21T23:47:32.361527+0000 (oldest deadline 2024-02-21T23:47:54.661354+0000) 2024-02-21T23:48:16.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:15 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:15.953+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:39.962203+0000 front 2024-02-21T23:47:39.962151+0000 (oldest deadline 2024-02-21T23:48:04.662264+0000) 2024-02-21T23:48:16.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:15 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:15.953+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:45.163315+0000 front 2024-02-21T23:47:45.163402+0000 (oldest deadline 2024-02-21T23:48:10.463241+0000) 2024-02-21T23:48:16.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:15 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:15.953+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-02-21T23:47:53.865163+0000 front 2024-02-21T23:47:53.865128+0000 (oldest deadline 2024-02-21T23:48:14.965109+0000) 2024-02-21T23:48:16.981 INFO:journalctl@ceph.osd.9.smithi090.stdout:Feb 21 23:48:16 smithi090 bash[100134]: ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-9 2024-02-21T23:48:17.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:16 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:16.844+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:48:17.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:16 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:16.844+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:48:17.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:16 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:16.844+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:48:17.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:16 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:16.844+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:38.458066+0000 front 2024-02-21T23:47:38.457888+0000 (oldest deadline 2024-02-21T23:48:00.757947+0000) 2024-02-21T23:48:17.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:16 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:16.844+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:46.559120+0000 front 2024-02-21T23:47:46.559318+0000 (oldest deadline 2024-02-21T23:48:08.259195+0000) 2024-02-21T23:48:17.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:16 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:16.844+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-02-21T23:47:50.560392+0000 front 2024-02-21T23:47:50.560281+0000 (oldest deadline 2024-02-21T23:48:16.460155+0000) 2024-02-21T23:48:17.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:16 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:16.915+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:48:17.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:16 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:16.915+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:48:17.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:16 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:16.915+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.160285+0000 front 2024-02-21T23:47:24.160271+0000 (oldest deadline 2024-02-21T23:47:48.860330+0000) 2024-02-21T23:48:17.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:16 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:16.915+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:32.361609+0000 front 2024-02-21T23:47:32.361527+0000 (oldest deadline 2024-02-21T23:47:54.661354+0000) 2024-02-21T23:48:17.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:16 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:16.915+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:39.962203+0000 front 2024-02-21T23:47:39.962151+0000 (oldest deadline 2024-02-21T23:48:04.662264+0000) 2024-02-21T23:48:17.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:16 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:16.915+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:45.163315+0000 front 2024-02-21T23:47:45.163402+0000 (oldest deadline 2024-02-21T23:48:10.463241+0000) 2024-02-21T23:48:17.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:16 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:16.915+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-02-21T23:47:53.865163+0000 front 2024-02-21T23:47:53.865128+0000 (oldest deadline 2024-02-21T23:48:14.965109+0000) 2024-02-21T23:48:17.669 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@osd.9.service' 2024-02-21T23:48:17.750 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-21T23:48:17.750 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-02-21T23:48:17.750 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-02-21T23:48:17.750 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-7ed81218-d112-11ee-95bd-87774f69a715@osd.10 2024-02-21T23:48:18.088 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:17 smithi176 systemd[1]: Stopping Ceph osd.10 for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:48:18.088 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:17 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:17.906+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:48:18.088 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:17 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:17.906+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:48:18.088 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:17 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:17.906+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.160285+0000 front 2024-02-21T23:47:24.160271+0000 (oldest deadline 2024-02-21T23:47:48.860330+0000) 2024-02-21T23:48:18.088 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:17 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:17.906+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:32.361609+0000 front 2024-02-21T23:47:32.361527+0000 (oldest deadline 2024-02-21T23:47:54.661354+0000) 2024-02-21T23:48:18.088 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:17 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:17.906+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:39.962203+0000 front 2024-02-21T23:47:39.962151+0000 (oldest deadline 2024-02-21T23:48:04.662264+0000) 2024-02-21T23:48:18.088 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:17 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:17.906+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:45.163315+0000 front 2024-02-21T23:47:45.163402+0000 (oldest deadline 2024-02-21T23:48:10.463241+0000) 2024-02-21T23:48:18.088 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:17 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:17.906+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-02-21T23:47:53.865163+0000 front 2024-02-21T23:47:53.865128+0000 (oldest deadline 2024-02-21T23:48:14.965109+0000) 2024-02-21T23:48:18.088 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:17 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:17.994+0000 7f910753f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-21T23:48:18.088 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:17 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:17.994+0000 7f910753f700 -1 osd.10 85 *** Got signal Terminated *** 2024-02-21T23:48:18.088 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:17 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:17.994+0000 7f910753f700 -1 osd.10 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-21T23:48:18.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:17 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:17.810+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:48:18.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:17 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:17.810+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:48:18.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:17 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:17.810+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:48:18.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:17 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:17.810+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:38.458066+0000 front 2024-02-21T23:47:38.457888+0000 (oldest deadline 2024-02-21T23:48:00.757947+0000) 2024-02-21T23:48:18.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:17 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:17.810+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:46.559120+0000 front 2024-02-21T23:47:46.559318+0000 (oldest deadline 2024-02-21T23:48:08.259195+0000) 2024-02-21T23:48:18.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:17 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:17.810+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-02-21T23:47:50.560392+0000 front 2024-02-21T23:47:50.560281+0000 (oldest deadline 2024-02-21T23:48:16.460155+0000) 2024-02-21T23:48:19.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:18 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:18.854+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:48:19.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:18 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:18.854+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:48:19.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:18 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:18.854+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:48:19.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:18 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:18.854+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:38.458066+0000 front 2024-02-21T23:47:38.457888+0000 (oldest deadline 2024-02-21T23:48:00.757947+0000) 2024-02-21T23:48:19.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:18 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:18.854+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:46.559120+0000 front 2024-02-21T23:47:46.559318+0000 (oldest deadline 2024-02-21T23:48:08.259195+0000) 2024-02-21T23:48:19.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:18 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:18.854+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-02-21T23:47:50.560392+0000 front 2024-02-21T23:47:50.560281+0000 (oldest deadline 2024-02-21T23:48:16.460155+0000) 2024-02-21T23:48:19.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:18 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:18.894+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:48:19.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:18 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:18.894+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:48:19.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:18 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:18.894+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.160285+0000 front 2024-02-21T23:47:24.160271+0000 (oldest deadline 2024-02-21T23:47:48.860330+0000) 2024-02-21T23:48:19.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:18 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:18.894+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:32.361609+0000 front 2024-02-21T23:47:32.361527+0000 (oldest deadline 2024-02-21T23:47:54.661354+0000) 2024-02-21T23:48:19.338 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:18 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:18.894+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:39.962203+0000 front 2024-02-21T23:47:39.962151+0000 (oldest deadline 2024-02-21T23:48:04.662264+0000) 2024-02-21T23:48:19.338 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:18 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:18.894+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:45.163315+0000 front 2024-02-21T23:47:45.163402+0000 (oldest deadline 2024-02-21T23:48:10.463241+0000) 2024-02-21T23:48:19.338 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:18 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:18.894+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-02-21T23:47:53.865163+0000 front 2024-02-21T23:47:53.865128+0000 (oldest deadline 2024-02-21T23:48:14.965109+0000) 2024-02-21T23:48:20.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:19 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:19.806+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:48:20.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:19 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:19.806+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:48:20.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:19 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:19.806+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:48:20.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:19 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:19.806+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:38.458066+0000 front 2024-02-21T23:47:38.457888+0000 (oldest deadline 2024-02-21T23:48:00.757947+0000) 2024-02-21T23:48:20.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:19 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:19.806+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:46.559120+0000 front 2024-02-21T23:47:46.559318+0000 (oldest deadline 2024-02-21T23:48:08.259195+0000) 2024-02-21T23:48:20.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:19 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:19.806+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-02-21T23:47:50.560392+0000 front 2024-02-21T23:47:50.560281+0000 (oldest deadline 2024-02-21T23:48:16.460155+0000) 2024-02-21T23:48:20.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:19 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:19.918+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:48:20.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:19 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:19.918+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:48:20.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:19 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:19.918+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.160285+0000 front 2024-02-21T23:47:24.160271+0000 (oldest deadline 2024-02-21T23:47:48.860330+0000) 2024-02-21T23:48:20.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:19 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:19.918+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:32.361609+0000 front 2024-02-21T23:47:32.361527+0000 (oldest deadline 2024-02-21T23:47:54.661354+0000) 2024-02-21T23:48:20.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:19 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:19.918+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:39.962203+0000 front 2024-02-21T23:47:39.962151+0000 (oldest deadline 2024-02-21T23:48:04.662264+0000) 2024-02-21T23:48:20.338 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:19 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:19.918+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:45.163315+0000 front 2024-02-21T23:47:45.163402+0000 (oldest deadline 2024-02-21T23:48:10.463241+0000) 2024-02-21T23:48:20.338 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:19 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:19.918+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-02-21T23:47:53.865163+0000 front 2024-02-21T23:47:53.865128+0000 (oldest deadline 2024-02-21T23:48:14.965109+0000) 2024-02-21T23:48:21.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:20 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:20.844+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:48:21.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:20 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:20.844+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:48:21.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:20 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:20.844+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:48:21.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:20 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:20.844+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:38.458066+0000 front 2024-02-21T23:47:38.457888+0000 (oldest deadline 2024-02-21T23:48:00.757947+0000) 2024-02-21T23:48:21.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:20 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:20.844+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:46.559120+0000 front 2024-02-21T23:47:46.559318+0000 (oldest deadline 2024-02-21T23:48:08.259195+0000) 2024-02-21T23:48:21.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:20 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:20.844+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-02-21T23:47:50.560392+0000 front 2024-02-21T23:47:50.560281+0000 (oldest deadline 2024-02-21T23:48:16.460155+0000) 2024-02-21T23:48:21.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:20 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:20.966+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:48:21.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:20 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:20.966+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:48:21.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:20 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:20.966+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.160285+0000 front 2024-02-21T23:47:24.160271+0000 (oldest deadline 2024-02-21T23:47:48.860330+0000) 2024-02-21T23:48:21.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:20 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:20.966+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:32.361609+0000 front 2024-02-21T23:47:32.361527+0000 (oldest deadline 2024-02-21T23:47:54.661354+0000) 2024-02-21T23:48:21.338 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:20 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:20.966+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:39.962203+0000 front 2024-02-21T23:47:39.962151+0000 (oldest deadline 2024-02-21T23:48:04.662264+0000) 2024-02-21T23:48:21.338 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:20 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:20.966+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:45.163315+0000 front 2024-02-21T23:47:45.163402+0000 (oldest deadline 2024-02-21T23:48:10.463241+0000) 2024-02-21T23:48:21.338 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:20 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:20.966+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-02-21T23:47:53.865163+0000 front 2024-02-21T23:47:53.865128+0000 (oldest deadline 2024-02-21T23:48:14.965109+0000) 2024-02-21T23:48:22.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:21 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:21.881+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:48:22.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:21 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:21.881+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:48:22.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:21 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:21.881+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:48:22.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:21 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:21.881+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:38.458066+0000 front 2024-02-21T23:47:38.457888+0000 (oldest deadline 2024-02-21T23:48:00.757947+0000) 2024-02-21T23:48:22.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:21 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:21.881+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:46.559120+0000 front 2024-02-21T23:47:46.559318+0000 (oldest deadline 2024-02-21T23:48:08.259195+0000) 2024-02-21T23:48:22.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:21 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:21.881+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-02-21T23:47:50.560392+0000 front 2024-02-21T23:47:50.560281+0000 (oldest deadline 2024-02-21T23:48:16.460155+0000) 2024-02-21T23:48:22.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:22 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:22.008+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:09.256148+0000 front 2024-02-21T23:47:09.256224+0000 (oldest deadline 2024-02-21T23:47:32.756155+0000) 2024-02-21T23:48:22.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:22 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:22.008+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:17.759039+0000 front 2024-02-21T23:47:17.758970+0000 (oldest deadline 2024-02-21T23:47:41.859172+0000) 2024-02-21T23:48:22.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:22 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:22.008+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.160285+0000 front 2024-02-21T23:47:24.160271+0000 (oldest deadline 2024-02-21T23:47:48.860330+0000) 2024-02-21T23:48:22.337 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:22 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:22.008+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-02-21T23:47:32.361609+0000 front 2024-02-21T23:47:32.361527+0000 (oldest deadline 2024-02-21T23:47:54.661354+0000) 2024-02-21T23:48:22.338 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:22 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:22.008+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:39.962203+0000 front 2024-02-21T23:47:39.962151+0000 (oldest deadline 2024-02-21T23:48:04.662264+0000) 2024-02-21T23:48:22.338 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:22 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:22.008+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:45.163315+0000 front 2024-02-21T23:47:45.163402+0000 (oldest deadline 2024-02-21T23:48:10.463241+0000) 2024-02-21T23:48:22.338 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:22 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10[76752]: 2024-02-21T23:48:22.008+0000 7f9102332700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-02-21T23:47:53.865163+0000 front 2024-02-21T23:47:53.865128+0000 (oldest deadline 2024-02-21T23:48:14.965109+0000) 2024-02-21T23:48:23.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:22 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:22.896+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:48:23.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:22 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:22.896+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:48:23.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:22 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:22.896+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:48:23.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:22 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:22.896+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:38.458066+0000 front 2024-02-21T23:47:38.457888+0000 (oldest deadline 2024-02-21T23:48:00.757947+0000) 2024-02-21T23:48:23.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:22 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:22.896+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:46.559120+0000 front 2024-02-21T23:47:46.559318+0000 (oldest deadline 2024-02-21T23:48:08.259195+0000) 2024-02-21T23:48:23.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:22 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:22.896+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-02-21T23:47:50.560392+0000 front 2024-02-21T23:47:50.560281+0000 (oldest deadline 2024-02-21T23:48:16.460155+0000) 2024-02-21T23:48:23.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:22 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:22.896+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6820 osd.7 since back 2024-02-21T23:47:56.460659+0000 front 2024-02-21T23:47:56.460716+0000 (oldest deadline 2024-02-21T23:48:22.360660+0000) 2024-02-21T23:48:23.837 INFO:journalctl@ceph.osd.10.smithi176.stdout:Feb 21 23:48:23 smithi176 bash[86058]: ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-10 2024-02-21T23:48:24.177 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:23 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:23.930+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:48:24.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:23 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:23.930+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:48:24.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:23 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:23.930+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:48:24.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:23 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:23.930+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:38.458066+0000 front 2024-02-21T23:47:38.457888+0000 (oldest deadline 2024-02-21T23:48:00.757947+0000) 2024-02-21T23:48:24.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:23 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:23.930+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:46.559120+0000 front 2024-02-21T23:47:46.559318+0000 (oldest deadline 2024-02-21T23:48:08.259195+0000) 2024-02-21T23:48:24.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:23 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:23.930+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-02-21T23:47:50.560392+0000 front 2024-02-21T23:47:50.560281+0000 (oldest deadline 2024-02-21T23:48:16.460155+0000) 2024-02-21T23:48:24.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:23 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:23.930+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6820 osd.7 since back 2024-02-21T23:47:56.460659+0000 front 2024-02-21T23:47:56.460716+0000 (oldest deadline 2024-02-21T23:48:22.360660+0000) 2024-02-21T23:48:24.627 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@osd.10.service' 2024-02-21T23:48:24.668 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-21T23:48:24.668 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-02-21T23:48:24.669 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-02-21T23:48:24.669 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-7ed81218-d112-11ee-95bd-87774f69a715@osd.11 2024-02-21T23:48:24.967 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:24 smithi178 systemd[1]: Stopping Ceph osd.11 for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:48:24.967 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:24 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:24.925+0000 7f3f04e30700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-21T23:48:24.967 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:24 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:24.925+0000 7f3f04e30700 -1 osd.11 85 *** Got signal Terminated *** 2024-02-21T23:48:24.967 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:24 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:24.925+0000 7f3f04e30700 -1 osd.11 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-21T23:48:25.428 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:24 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:24.979+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:48:25.428 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:24 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:24.979+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:48:25.428 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:24 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:24.979+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:48:25.428 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:24 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:24.979+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:38.458066+0000 front 2024-02-21T23:47:38.457888+0000 (oldest deadline 2024-02-21T23:48:00.757947+0000) 2024-02-21T23:48:25.428 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:24 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:24.979+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:46.559120+0000 front 2024-02-21T23:47:46.559318+0000 (oldest deadline 2024-02-21T23:48:08.259195+0000) 2024-02-21T23:48:25.428 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:24 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:24.979+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-02-21T23:47:50.560392+0000 front 2024-02-21T23:47:50.560281+0000 (oldest deadline 2024-02-21T23:48:16.460155+0000) 2024-02-21T23:48:25.428 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:24 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:24.979+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6820 osd.7 since back 2024-02-21T23:47:56.460659+0000 front 2024-02-21T23:47:56.460716+0000 (oldest deadline 2024-02-21T23:48:22.360660+0000) 2024-02-21T23:48:26.427 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:26 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:26.009+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:48:26.428 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:26 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:26.009+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:48:26.428 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:26 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:26.009+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:48:26.428 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:26 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:26.009+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:38.458066+0000 front 2024-02-21T23:47:38.457888+0000 (oldest deadline 2024-02-21T23:48:00.757947+0000) 2024-02-21T23:48:26.428 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:26 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:26.009+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:46.559120+0000 front 2024-02-21T23:47:46.559318+0000 (oldest deadline 2024-02-21T23:48:08.259195+0000) 2024-02-21T23:48:26.428 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:26 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:26.009+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-02-21T23:47:50.560392+0000 front 2024-02-21T23:47:50.560281+0000 (oldest deadline 2024-02-21T23:48:16.460155+0000) 2024-02-21T23:48:26.428 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:26 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:26.009+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6820 osd.7 since back 2024-02-21T23:47:56.460659+0000 front 2024-02-21T23:47:56.460716+0000 (oldest deadline 2024-02-21T23:48:22.360660+0000) 2024-02-21T23:48:27.428 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:27 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:27.008+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:48:27.428 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:27 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:27.008+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:48:27.428 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:27 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:27.008+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:48:27.428 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:27 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:27.008+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:38.458066+0000 front 2024-02-21T23:47:38.457888+0000 (oldest deadline 2024-02-21T23:48:00.757947+0000) 2024-02-21T23:48:27.428 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:27 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:27.008+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:46.559120+0000 front 2024-02-21T23:47:46.559318+0000 (oldest deadline 2024-02-21T23:48:08.259195+0000) 2024-02-21T23:48:27.428 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:27 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:27.008+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-02-21T23:47:50.560392+0000 front 2024-02-21T23:47:50.560281+0000 (oldest deadline 2024-02-21T23:48:16.460155+0000) 2024-02-21T23:48:27.428 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:27 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:27.008+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6820 osd.7 since back 2024-02-21T23:47:56.460659+0000 front 2024-02-21T23:47:56.460716+0000 (oldest deadline 2024-02-21T23:48:22.360660+0000) 2024-02-21T23:48:28.428 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:28 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:28.021+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:48:28.428 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:28 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:28.021+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:48:28.428 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:28 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:28.021+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:48:28.428 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:28 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:28.021+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:38.458066+0000 front 2024-02-21T23:47:38.457888+0000 (oldest deadline 2024-02-21T23:48:00.757947+0000) 2024-02-21T23:48:28.428 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:28 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:28.021+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:46.559120+0000 front 2024-02-21T23:47:46.559318+0000 (oldest deadline 2024-02-21T23:48:08.259195+0000) 2024-02-21T23:48:28.428 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:28 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:28.021+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-02-21T23:47:50.560392+0000 front 2024-02-21T23:47:50.560281+0000 (oldest deadline 2024-02-21T23:48:16.460155+0000) 2024-02-21T23:48:28.428 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:28 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:28.021+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6820 osd.7 since back 2024-02-21T23:47:56.460659+0000 front 2024-02-21T23:47:56.460716+0000 (oldest deadline 2024-02-21T23:48:22.360660+0000) 2024-02-21T23:48:29.428 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:29 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:29.022+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-02-21T23:47:10.651978+0000 front 2024-02-21T23:47:10.652061+0000 (oldest deadline 2024-02-21T23:47:34.752115+0000) 2024-02-21T23:48:29.428 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:29 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:29.022+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6804 osd.1 since back 2024-02-21T23:47:16.953822+0000 front 2024-02-21T23:47:16.953835+0000 (oldest deadline 2024-02-21T23:47:39.253960+0000) 2024-02-21T23:48:29.428 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:29 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:29.022+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-02-21T23:47:24.356089+0000 front 2024-02-21T23:47:24.356062+0000 (oldest deadline 2024-02-21T23:47:50.256301+0000) 2024-02-21T23:48:29.428 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:29 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:29.022+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-02-21T23:47:38.458066+0000 front 2024-02-21T23:47:38.457888+0000 (oldest deadline 2024-02-21T23:48:00.757947+0000) 2024-02-21T23:48:29.428 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:29 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:29.022+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-02-21T23:47:46.559120+0000 front 2024-02-21T23:47:46.559318+0000 (oldest deadline 2024-02-21T23:48:08.259195+0000) 2024-02-21T23:48:29.428 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:29 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:29.022+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-02-21T23:47:50.560392+0000 front 2024-02-21T23:47:50.560281+0000 (oldest deadline 2024-02-21T23:48:16.460155+0000) 2024-02-21T23:48:29.429 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:29 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11[79301]: 2024-02-21T23:48:29.022+0000 7f3eff422700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.176:6820 osd.7 since back 2024-02-21T23:47:56.460659+0000 front 2024-02-21T23:47:56.460716+0000 (oldest deadline 2024-02-21T23:48:22.360660+0000) 2024-02-21T23:48:30.928 INFO:journalctl@ceph.osd.11.smithi178.stdout:Feb 21 23:48:30 smithi178 bash[88152]: ceph-7ed81218-d112-11ee-95bd-87774f69a715-osd-11 2024-02-21T23:48:31.964 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@osd.11.service' 2024-02-21T23:48:32.005 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-21T23:48:32.005 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-02-21T23:48:32.006 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-02-21T23:48:32.006 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-7ed81218-d112-11ee-95bd-87774f69a715@mds.a 2024-02-21T23:48:32.481 INFO:journalctl@ceph.mds.a.smithi090.stdout:Feb 21 23:48:32 smithi090 systemd[1]: Stopping Ceph mds.a for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:48:32.481 INFO:journalctl@ceph.mds.a.smithi090.stdout:Feb 21 23:48:32 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-a[95168]: 2024-02-21T23:48:32.251+0000 7f08dbeda700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-21T23:48:32.481 INFO:journalctl@ceph.mds.a.smithi090.stdout:Feb 21 23:48:32 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-a[95168]: 2024-02-21T23:48:32.251+0000 7f08dbeda700 -1 mds.a *** got signal Terminated *** 2024-02-21T23:48:35.731 INFO:journalctl@ceph.mds.a.smithi090.stdout:Feb 21 23:48:35 smithi090 bash[100398]: ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-a 2024-02-21T23:48:35.777 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@mds.a.service' 2024-02-21T23:48:35.813 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-21T23:48:35.813 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-02-21T23:48:35.813 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-02-21T23:48:35.813 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-7ed81218-d112-11ee-95bd-87774f69a715@mds.d 2024-02-21T23:48:36.081 INFO:journalctl@ceph.mds.d.smithi090.stdout:Feb 21 23:48:35 smithi090 systemd[1]: Stopping Ceph mds.d for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:48:36.481 INFO:journalctl@ceph.mds.d.smithi090.stdout:Feb 21 23:48:36 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-d[94250]: 2024-02-21T23:48:36.091+0000 7f8c15258700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-21T23:48:36.481 INFO:journalctl@ceph.mds.d.smithi090.stdout:Feb 21 23:48:36 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-d[94250]: 2024-02-21T23:48:36.091+0000 7f8c15258700 -1 mds.d *** got signal Terminated *** 2024-02-21T23:48:38.979 INFO:journalctl@ceph.mds.d.smithi090.stdout:Feb 21 23:48:38 smithi090 bash[100537]: ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-d 2024-02-21T23:48:38.984 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@mds.d.service' 2024-02-21T23:48:39.017 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-21T23:48:39.018 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-02-21T23:48:39.018 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-02-21T23:48:39.018 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-7ed81218-d112-11ee-95bd-87774f69a715@mds.g 2024-02-21T23:48:39.288 INFO:journalctl@ceph.mds.g.smithi090.stdout:Feb 21 23:48:39 smithi090 systemd[1]: Stopping Ceph mds.g for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:48:39.731 INFO:journalctl@ceph.mds.g.smithi090.stdout:Feb 21 23:48:39 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-g[94779]: 2024-02-21T23:48:39.299+0000 7f7c2f816700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-21T23:48:39.731 INFO:journalctl@ceph.mds.g.smithi090.stdout:Feb 21 23:48:39 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-g[94779]: 2024-02-21T23:48:39.299+0000 7f7c2f816700 -1 mds.g *** got signal Terminated *** 2024-02-21T23:48:41.910 INFO:journalctl@ceph.mds.g.smithi090.stdout:Feb 21 23:48:41 smithi090 bash[100675]: ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-g 2024-02-21T23:48:41.917 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@mds.g.service' 2024-02-21T23:48:41.951 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-21T23:48:41.951 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-02-21T23:48:41.951 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-02-21T23:48:41.952 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-7ed81218-d112-11ee-95bd-87774f69a715@mds.j 2024-02-21T23:48:42.205 INFO:journalctl@ceph.mds.j.smithi090.stdout:Feb 21 23:48:42 smithi090 systemd[1]: Stopping Ceph mds.j for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:48:42.481 INFO:journalctl@ceph.mds.j.smithi090.stdout:Feb 21 23:48:42 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-j[93864]: 2024-02-21T23:48:42.216+0000 7f844f6f5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-21T23:48:42.481 INFO:journalctl@ceph.mds.j.smithi090.stdout:Feb 21 23:48:42 smithi090 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-j[93864]: 2024-02-21T23:48:42.216+0000 7f844f6f5700 -1 mds.j *** got signal Terminated *** 2024-02-21T23:48:45.221 INFO:journalctl@ceph.mds.j.smithi090.stdout:Feb 21 23:48:44 smithi090 bash[100814]: ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-j 2024-02-21T23:48:45.226 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@mds.j.service' 2024-02-21T23:48:45.260 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-21T23:48:45.261 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-02-21T23:48:45.261 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-02-21T23:48:45.261 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-7ed81218-d112-11ee-95bd-87774f69a715@mds.b 2024-02-21T23:48:45.588 INFO:journalctl@ceph.mds.b.smithi176.stdout:Feb 21 23:48:45 smithi176 systemd[1]: Stopping Ceph mds.b for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:48:45.588 INFO:journalctl@ceph.mds.b.smithi176.stdout:Feb 21 23:48:45 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-b[83617]: 2024-02-21T23:48:45.506+0000 7fa8f4d2d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-21T23:48:45.588 INFO:journalctl@ceph.mds.b.smithi176.stdout:Feb 21 23:48:45 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-b[83617]: 2024-02-21T23:48:45.506+0000 7fa8f4d2d700 -1 mds.b *** got signal Terminated *** 2024-02-21T23:48:51.088 INFO:journalctl@ceph.mds.b.smithi176.stdout:Feb 21 23:48:50 smithi176 bash[86322]: ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-b 2024-02-21T23:48:51.152 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@mds.b.service' 2024-02-21T23:48:51.190 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-21T23:48:51.191 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-02-21T23:48:51.191 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-02-21T23:48:51.191 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-7ed81218-d112-11ee-95bd-87774f69a715@mds.e 2024-02-21T23:48:51.452 INFO:journalctl@ceph.mds.e.smithi176.stdout:Feb 21 23:48:51 smithi176 systemd[1]: Stopping Ceph mds.e for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:48:51.838 INFO:journalctl@ceph.mds.e.smithi176.stdout:Feb 21 23:48:51 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-e[82935]: 2024-02-21T23:48:51.463+0000 7f86bd152700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-21T23:48:51.838 INFO:journalctl@ceph.mds.e.smithi176.stdout:Feb 21 23:48:51 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-e[82935]: 2024-02-21T23:48:51.463+0000 7f86bd152700 -1 mds.e *** got signal Terminated *** 2024-02-21T23:48:55.838 INFO:journalctl@ceph.mds.e.smithi176.stdout:Feb 21 23:48:55 smithi176 bash[86461]: ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-e 2024-02-21T23:48:55.867 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@mds.e.service' 2024-02-21T23:48:55.901 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-21T23:48:55.901 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-02-21T23:48:55.901 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-02-21T23:48:55.901 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-7ed81218-d112-11ee-95bd-87774f69a715@mds.h 2024-02-21T23:48:56.152 INFO:journalctl@ceph.mds.h.smithi176.stdout:Feb 21 23:48:55 smithi176 systemd[1]: Stopping Ceph mds.h for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:48:56.588 INFO:journalctl@ceph.mds.h.smithi176.stdout:Feb 21 23:48:56 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-h[83278]: 2024-02-21T23:48:56.163+0000 7f93e440a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-21T23:48:56.588 INFO:journalctl@ceph.mds.h.smithi176.stdout:Feb 21 23:48:56 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-h[83278]: 2024-02-21T23:48:56.163+0000 7f93e440a700 -1 mds.h *** got signal Terminated *** 2024-02-21T23:48:58.092 INFO:journalctl@ceph.mds.h.smithi176.stdout:Feb 21 23:48:57 smithi176 bash[86599]: ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-h 2024-02-21T23:48:58.130 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@mds.h.service' 2024-02-21T23:48:58.163 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-21T23:48:58.164 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-02-21T23:48:58.164 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-02-21T23:48:58.164 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-7ed81218-d112-11ee-95bd-87774f69a715@mds.k 2024-02-21T23:48:58.418 INFO:journalctl@ceph.mds.k.smithi176.stdout:Feb 21 23:48:58 smithi176 systemd[1]: Stopping Ceph mds.k for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:48:58.838 INFO:journalctl@ceph.mds.k.smithi176.stdout:Feb 21 23:48:58 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-k[82592]: 2024-02-21T23:48:58.428+0000 7fe345d1e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-21T23:48:58.838 INFO:journalctl@ceph.mds.k.smithi176.stdout:Feb 21 23:48:58 smithi176 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-k[82592]: 2024-02-21T23:48:58.428+0000 7fe345d1e700 -1 mds.k *** got signal Terminated *** 2024-02-21T23:49:02.838 INFO:journalctl@ceph.mds.k.smithi176.stdout:Feb 21 23:49:02 smithi176 bash[86738]: ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-k 2024-02-21T23:49:02.895 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@mds.k.service' 2024-02-21T23:49:02.929 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-21T23:49:02.929 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-02-21T23:49:02.930 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-02-21T23:49:02.930 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-7ed81218-d112-11ee-95bd-87774f69a715@mds.c 2024-02-21T23:49:03.429 INFO:journalctl@ceph.mds.c.smithi178.stdout:Feb 21 23:49:02 smithi178 systemd[1]: Stopping Ceph mds.c for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:49:03.429 INFO:journalctl@ceph.mds.c.smithi178.stdout:Feb 21 23:49:03 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-c[85367]: 2024-02-21T23:49:03.192+0000 7fe50a591700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-21T23:49:03.429 INFO:journalctl@ceph.mds.c.smithi178.stdout:Feb 21 23:49:03 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-c[85367]: 2024-02-21T23:49:03.192+0000 7fe50a591700 -1 mds.c *** got signal Terminated *** 2024-02-21T23:49:07.679 INFO:journalctl@ceph.mds.c.smithi178.stdout:Feb 21 23:49:07 smithi178 bash[88415]: ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-c 2024-02-21T23:49:07.832 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@mds.c.service' 2024-02-21T23:49:07.870 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-21T23:49:07.870 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-02-21T23:49:07.870 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-02-21T23:49:07.870 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-7ed81218-d112-11ee-95bd-87774f69a715@mds.f 2024-02-21T23:49:08.147 INFO:journalctl@ceph.mds.f.smithi178.stdout:Feb 21 23:49:07 smithi178 systemd[1]: Stopping Ceph mds.f for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:49:08.428 INFO:journalctl@ceph.mds.f.smithi178.stdout:Feb 21 23:49:08 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-f[85024]: 2024-02-21T23:49:08.158+0000 7f7e4e624700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-21T23:49:08.429 INFO:journalctl@ceph.mds.f.smithi178.stdout:Feb 21 23:49:08 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-f[85024]: 2024-02-21T23:49:08.158+0000 7f7e4e624700 -1 mds.f *** got signal Terminated *** 2024-02-21T23:49:10.916 INFO:journalctl@ceph.mds.f.smithi178.stdout:Feb 21 23:49:10 smithi178 bash[88553]: ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-f 2024-02-21T23:49:10.921 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@mds.f.service' 2024-02-21T23:49:10.956 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-21T23:49:10.956 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-02-21T23:49:10.956 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-02-21T23:49:10.956 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-7ed81218-d112-11ee-95bd-87774f69a715@mds.i 2024-02-21T23:49:11.179 INFO:journalctl@ceph.mds.i.smithi178.stdout:Feb 21 23:49:11 smithi178 systemd[1]: Stopping Ceph mds.i for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:49:11.679 INFO:journalctl@ceph.mds.i.smithi178.stdout:Feb 21 23:49:11 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-i[85706]: 2024-02-21T23:49:11.244+0000 7fa73ad34700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-21T23:49:11.679 INFO:journalctl@ceph.mds.i.smithi178.stdout:Feb 21 23:49:11 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-i[85706]: 2024-02-21T23:49:11.244+0000 7fa73ad34700 -1 mds.i *** got signal Terminated *** 2024-02-21T23:49:15.929 INFO:journalctl@ceph.mds.i.smithi178.stdout:Feb 21 23:49:15 smithi178 bash[88691]: ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-i 2024-02-21T23:49:15.977 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@mds.i.service' 2024-02-21T23:49:16.011 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-21T23:49:16.012 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-02-21T23:49:16.012 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-02-21T23:49:16.012 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-7ed81218-d112-11ee-95bd-87774f69a715@mds.l 2024-02-21T23:49:16.287 INFO:journalctl@ceph.mds.l.smithi178.stdout:Feb 21 23:49:16 smithi178 systemd[1]: Stopping Ceph mds.l for 7ed81218-d112-11ee-95bd-87774f69a715... 2024-02-21T23:49:16.679 INFO:journalctl@ceph.mds.l.smithi178.stdout:Feb 21 23:49:16 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-l[84682]: 2024-02-21T23:49:16.298+0000 7ff8cac65700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-21T23:49:16.679 INFO:journalctl@ceph.mds.l.smithi178.stdout:Feb 21 23:49:16 smithi178 ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-l[84682]: 2024-02-21T23:49:16.298+0000 7ff8cac65700 -1 mds.l *** got signal Terminated *** 2024-02-21T23:49:17.679 INFO:journalctl@ceph.mds.l.smithi178.stdout:Feb 21 23:49:17 smithi178 bash[88830]: ceph-7ed81218-d112-11ee-95bd-87774f69a715-mds-l 2024-02-21T23:49:17.858 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ed81218-d112-11ee-95bd-87774f69a715@mds.l.service' 2024-02-21T23:49:17.891 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-21T23:49:17.891 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-02-21T23:49:17.892 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7ed81218-d112-11ee-95bd-87774f69a715 --force --keep-logs 2024-02-21T23:49:18.078 INFO:teuthology.orchestra.run.smithi090.stdout:Deleting cluster with fsid: 7ed81218-d112-11ee-95bd-87774f69a715 2024-02-21T23:49:19.450 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7ed81218-d112-11ee-95bd-87774f69a715 --force --keep-logs 2024-02-21T23:49:19.641 INFO:teuthology.orchestra.run.smithi176.stdout:Deleting cluster with fsid: 7ed81218-d112-11ee-95bd-87774f69a715 2024-02-21T23:49:21.026 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7ed81218-d112-11ee-95bd-87774f69a715 --force --keep-logs 2024-02-21T23:49:21.212 INFO:teuthology.orchestra.run.smithi178.stdout:Deleting cluster with fsid: 7ed81218-d112-11ee-95bd-87774f69a715 2024-02-21T23:49:22.603 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-21T23:49:22.639 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-21T23:49:22.677 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-21T23:49:22.708 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-21T23:49:22.710 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-21_23:06:32-fs-wip-yuri10-testing-2024-02-21-1249-reef-distro-default-smithi/7570183/remote/smithi090/crash 2024-02-21T23:49:22.711 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/crash -- . 2024-02-21T23:49:22.758 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/crash: Cannot open: No such file or directory 2024-02-21T23:49:22.758 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2024-02-21T23:49:22.760 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-21_23:06:32-fs-wip-yuri10-testing-2024-02-21-1249-reef-distro-default-smithi/7570183/remote/smithi176/crash 2024-02-21T23:49:22.760 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/crash -- . 2024-02-21T23:49:22.805 INFO:teuthology.orchestra.run.smithi176.stderr:tar: /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/crash: Cannot open: No such file or directory 2024-02-21T23:49:22.806 INFO:teuthology.orchestra.run.smithi176.stderr:tar: Error is not recoverable: exiting now 2024-02-21T23:49:22.807 DEBUG:teuthology.misc:Transferring archived files from smithi178:/var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-21_23:06:32-fs-wip-yuri10-testing-2024-02-21-1249-reef-distro-default-smithi/7570183/remote/smithi178/crash 2024-02-21T23:49:22.808 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/crash -- . 2024-02-21T23:49:22.843 INFO:teuthology.orchestra.run.smithi178.stderr:tar: /var/lib/ceph/7ed81218-d112-11ee-95bd-87774f69a715/crash: Cannot open: No such file or directory 2024-02-21T23:49:22.843 INFO:teuthology.orchestra.run.smithi178.stderr:tar: Error is not recoverable: exiting now 2024-02-21T23:49:22.844 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-21T23:49:22.845 DEBUG:teuthology.orchestra.run.smithi090:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/7ed81218-d112-11ee-95bd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-02-21T23:49:22.880 INFO:teuthology.orchestra.run.smithi090.stderr:grep: /var/log/ceph/7ed81218-d112-11ee-95bd-87774f69a715/ceph.log: No such file or directory 2024-02-21T23:49:22.882 INFO:tasks.cephadm:Compressing logs... 2024-02-21T23:49:22.882 DEBUG:teuthology.orchestra.run.smithi090:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-21T23:49:22.925 DEBUG:teuthology.orchestra.run.smithi176:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-21T23:49:22.928 DEBUG:teuthology.orchestra.run.smithi178:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-21T23:49:22.952 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-02-21T23:49:22.955 INFO:teuthology.orchestra.run.smithi178.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-02-21T23:49:22.958 INFO:teuthology.orchestra.run.smithi176.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-02-21T23:49:24.072 INFO:tasks.cephadm:Archiving logs... 2024-02-21T23:49:24.073 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-21_23:06:32-fs-wip-yuri10-testing-2024-02-21-1249-reef-distro-default-smithi/7570183/remote/smithi090/log 2024-02-21T23:49:24.074 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-21T23:49:24.324 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-21_23:06:32-fs-wip-yuri10-testing-2024-02-21-1249-reef-distro-default-smithi/7570183/remote/smithi176/log 2024-02-21T23:49:24.325 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-21T23:49:24.494 DEBUG:teuthology.misc:Transferring archived files from smithi178:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-21_23:06:32-fs-wip-yuri10-testing-2024-02-21-1249-reef-distro-default-smithi/7570183/remote/smithi178/log 2024-02-21T23:49:24.495 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-21T23:49:24.637 INFO:tasks.cephadm:Removing cluster... 2024-02-21T23:49:24.637 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7ed81218-d112-11ee-95bd-87774f69a715 --force 2024-02-21T23:49:24.807 INFO:teuthology.orchestra.run.smithi090.stdout:Deleting cluster with fsid: 7ed81218-d112-11ee-95bd-87774f69a715 2024-02-21T23:49:25.034 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7ed81218-d112-11ee-95bd-87774f69a715 --force 2024-02-21T23:49:25.212 INFO:teuthology.orchestra.run.smithi176.stdout:Deleting cluster with fsid: 7ed81218-d112-11ee-95bd-87774f69a715 2024-02-21T23:49:25.447 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7ed81218-d112-11ee-95bd-87774f69a715 --force 2024-02-21T23:49:25.639 INFO:teuthology.orchestra.run.smithi178.stdout:Deleting cluster with fsid: 7ed81218-d112-11ee-95bd-87774f69a715 2024-02-21T23:49:25.873 INFO:tasks.cephadm:Removing cephadm ... 2024-02-21T23:49:25.874 DEBUG:teuthology.orchestra.run.smithi090:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-21T23:49:25.895 DEBUG:teuthology.orchestra.run.smithi176:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-21T23:49:25.915 DEBUG:teuthology.orchestra.run.smithi178:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-21T23:49:25.932 INFO:tasks.cephadm:Teardown complete 2024-02-21T23:49:25.933 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c1dc18f444603397768fcb7c34a8488b10284df6/qa/tasks/cephadm.py", line 1714, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_c1dc18f444603397768fcb7c34a8488b10284df6/qa/tasks/cephadm.py", line 1629, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c1dc18f444603397768fcb7c34a8488b10284df6/qa/tasks/cephadm.py", line 1051, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/github.com_ceph_ceph-c_c1dc18f444603397768fcb7c34a8488b10284df6/qa/tasks/cephfs/filesystem.py", line 532, in __init__ self.create() File "/home/teuthworker/src/github.com_ceph_ceph-c_c1dc18f444603397768fcb7c34a8488b10284df6/qa/tasks/cephfs/filesystem.py", line 762, in create self.run_ceph_cmd('fs', 'subvolumegroup', 'create', self.name, 'qa') AttributeError: 'Filesystem' object has no attribute 'run_ceph_cmd' 2024-02-21T23:49:26.160 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=9b27f320912d4810b1e4c3366f516ad1 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c1dc18f444603397768fcb7c34a8488b10284df6/qa/tasks/cephadm.py", line 1714, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_c1dc18f444603397768fcb7c34a8488b10284df6/qa/tasks/cephadm.py", line 1629, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c1dc18f444603397768fcb7c34a8488b10284df6/qa/tasks/cephadm.py", line 1051, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/github.com_ceph_ceph-c_c1dc18f444603397768fcb7c34a8488b10284df6/qa/tasks/cephfs/filesystem.py", line 532, in __init__ self.create() File "/home/teuthworker/src/github.com_ceph_ceph-c_c1dc18f444603397768fcb7c34a8488b10284df6/qa/tasks/cephfs/filesystem.py", line 762, in create self.run_ceph_cmd('fs', 'subvolumegroup', 'create', self.name, 'qa') AttributeError: 'Filesystem' object has no attribute 'run_ceph_cmd' 2024-02-21T23:49:26.164 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-21T23:49:26.173 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-21T23:49:26.215 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-21T23:49:26.216 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-21T23:49:26.230 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-21T23:49:26.234 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-21T23:49:26.342 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-21T23:49:26.350 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-21T23:49:26.351 DEBUG:teuthology.orchestra.run.smithi090:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-21T23:49:26.354 DEBUG:teuthology.orchestra.run.smithi176:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-21T23:49:26.356 DEBUG:teuthology.orchestra.run.smithi178:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-21T23:49:26.370 INFO:teuthology.orchestra.run.smithi090.stderr:bash: ntpq: command not found 2024-02-21T23:49:26.371 INFO:teuthology.orchestra.run.smithi176.stderr:bash: ntpq: command not found 2024-02-21T23:49:26.372 INFO:teuthology.orchestra.run.smithi178.stderr:bash: ntpq: command not found 2024-02-21T23:49:26.375 INFO:teuthology.orchestra.run.smithi090.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-21T23:49:26.375 INFO:teuthology.orchestra.run.smithi090.stdout:=============================================================================== 2024-02-21T23:49:26.375 INFO:teuthology.orchestra.run.smithi090.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 8 +388us[ +388us] +/- 78ms 2024-02-21T23:49:26.375 INFO:teuthology.orchestra.run.smithi090.stdout:^* hv02.front.sepia.ceph.com 3 6 377 9 +130us[ +135us] +/- 63ms 2024-02-21T23:49:26.375 INFO:teuthology.orchestra.run.smithi090.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 8 -410us[ -410us] +/- 63ms 2024-02-21T23:49:26.376 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-02-21T23:49:26.377 INFO:teuthology.orchestra.run.smithi178.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-21T23:49:26.378 INFO:teuthology.orchestra.run.smithi178.stdout:=============================================================================== 2024-02-21T23:49:26.378 INFO:teuthology.orchestra.run.smithi178.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 8 +384us[ +384us] +/- 78ms 2024-02-21T23:49:26.378 INFO:teuthology.orchestra.run.smithi178.stdout:^* hv02.front.sepia.ceph.com 3 6 377 11 +135us[ +143us] +/- 63ms 2024-02-21T23:49:26.378 INFO:teuthology.orchestra.run.smithi178.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 9 -411us[ -411us] +/- 62ms 2024-02-21T23:49:26.378 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-02-21T23:49:26.378 INFO:teuthology.orchestra.run.smithi176.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-21T23:49:26.378 INFO:teuthology.orchestra.run.smithi176.stdout:=============================================================================== 2024-02-21T23:49:26.379 INFO:teuthology.orchestra.run.smithi176.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 8 +385us[ +385us] +/- 78ms 2024-02-21T23:49:26.379 INFO:teuthology.orchestra.run.smithi176.stdout:^* hv02.front.sepia.ceph.com 3 6 377 9 +124us[ +127us] +/- 63ms 2024-02-21T23:49:26.379 INFO:teuthology.orchestra.run.smithi176.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 9 -401us[ -398us] +/- 62ms 2024-02-21T23:49:26.379 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-02-21T23:49:26.379 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-21T23:49:26.388 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-21T23:49:26.388 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-21T23:49:26.407 DEBUG:teuthology.orchestra.run.smithi090:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-21T23:49:26.492 DEBUG:teuthology.orchestra.run.smithi176:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-21T23:49:26.562 DEBUG:teuthology.orchestra.run.smithi178:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-21T23:49:26.629 DEBUG:teuthology.orchestra.run.smithi090:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-21T23:49:26.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-21T23:49:26.661 DEBUG:teuthology.orchestra.run.smithi176:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-21T23:49:26.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-21T23:49:26.693 DEBUG:teuthology.orchestra.run.smithi178:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-21T23:49:26.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-21T23:49:26.723 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-21T23:49:26.733 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-21T23:49:26.783 INFO:teuthology.task.internal:Duration was 1413.084720 seconds 2024-02-21T23:49:26.783 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-21T23:49:26.816 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-21T23:49:26.816 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-21T23:49:26.819 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-21T23:49:26.821 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-21T23:49:26.866 INFO:teuthology.orchestra.run.smithi090.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-21T23:49:26.867 INFO:teuthology.orchestra.run.smithi176.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-21T23:49:26.868 INFO:teuthology.orchestra.run.smithi178.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-21T23:49:27.353 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-21T23:49:27.353 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi090.front.sepia.ceph.com 2024-02-21T23:49:27.353 DEBUG:teuthology.orchestra.run.smithi090:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-21T23:49:27.380 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi176.front.sepia.ceph.com 2024-02-21T23:49:27.380 DEBUG:teuthology.orchestra.run.smithi176:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-21T23:49:27.403 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi178.front.sepia.ceph.com 2024-02-21T23:49:27.404 DEBUG:teuthology.orchestra.run.smithi178:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-21T23:49:27.426 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-21T23:49:27.427 DEBUG:teuthology.orchestra.run.smithi090:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-21T23:49:27.429 DEBUG:teuthology.orchestra.run.smithi176:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-21T23:49:27.446 DEBUG:teuthology.orchestra.run.smithi178:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-21T23:49:27.518 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-21T23:49:27.518 DEBUG:teuthology.orchestra.run.smithi090:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-21T23:49:27.521 DEBUG:teuthology.orchestra.run.smithi176:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-21T23:49:27.523 DEBUG:teuthology.orchestra.run.smithi178:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-21T23:49:27.621 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-21T23:49:27.631 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-21T23:49:27.631 DEBUG:teuthology.orchestra.run.smithi090:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-21T23:49:27.658 DEBUG:teuthology.orchestra.run.smithi176:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-21T23:49:27.686 DEBUG:teuthology.orchestra.run.smithi178:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-21T23:49:27.714 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-21T23:49:27.727 DEBUG:teuthology.orchestra.run.smithi090:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-21T23:49:27.730 DEBUG:teuthology.orchestra.run.smithi176:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-21T23:49:27.732 DEBUG:teuthology.orchestra.run.smithi178:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-21T23:49:27.754 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern = core 2024-02-21T23:49:27.757 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern = core 2024-02-21T23:49:27.780 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern = core 2024-02-21T23:49:27.804 DEBUG:teuthology.orchestra.run.smithi090:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-21T23:49:27.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-21T23:49:27.826 DEBUG:teuthology.orchestra.run.smithi176:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-21T23:49:27.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-21T23:49:27.844 DEBUG:teuthology.orchestra.run.smithi178:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-21T23:49:27.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-21T23:49:27.860 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-21T23:49:27.872 INFO:teuthology.task.internal:Transferring archived files... 2024-02-21T23:49:27.873 DEBUG:teuthology.misc:Transferring archived files from smithi090:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-21_23:06:32-fs-wip-yuri10-testing-2024-02-21-1249-reef-distro-default-smithi/7570183/remote/smithi090 2024-02-21T23:49:27.873 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-21T23:49:27.914 DEBUG:teuthology.misc:Transferring archived files from smithi176:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-21_23:06:32-fs-wip-yuri10-testing-2024-02-21-1249-reef-distro-default-smithi/7570183/remote/smithi176 2024-02-21T23:49:27.914 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-21T23:49:27.954 DEBUG:teuthology.misc:Transferring archived files from smithi178:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-21_23:06:32-fs-wip-yuri10-testing-2024-02-21-1249-reef-distro-default-smithi/7570183/remote/smithi178 2024-02-21T23:49:27.954 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-21T23:49:27.997 INFO:teuthology.task.internal:Removing archive directory... 2024-02-21T23:49:27.997 DEBUG:teuthology.orchestra.run.smithi090:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-21T23:49:27.999 DEBUG:teuthology.orchestra.run.smithi176:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-21T23:49:28.001 DEBUG:teuthology.orchestra.run.smithi178:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-21T23:49:28.047 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-21T23:49:28.058 INFO:teuthology.task.internal:Not uploading archives. 2024-02-21T23:49:28.058 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-21T23:49:28.103 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-21T23:49:28.103 DEBUG:teuthology.orchestra.run.smithi090:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-21T23:49:28.106 DEBUG:teuthology.orchestra.run.smithi176:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-21T23:49:28.108 DEBUG:teuthology.orchestra.run.smithi178:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-21T23:49:28.120 INFO:teuthology.orchestra.run.smithi090.stdout: 262183 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 21 23:49 /home/ubuntu/cephtest 2024-02-21T23:49:28.122 INFO:teuthology.orchestra.run.smithi176.stdout: 262157 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 21 23:49 /home/ubuntu/cephtest 2024-02-21T23:49:28.125 INFO:teuthology.orchestra.run.smithi178.stdout: 262157 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 21 23:49 /home/ubuntu/cephtest 2024-02-21T23:49:28.126 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-21T23:49:28.141 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-21T23:49:28.209 INFO:teuthology.nuke:Checking targets against current locks 2024-02-21T23:49:28.237 DEBUG:teuthology.nuke:shortname: smithi090 2024-02-21T23:49:28.237 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-21T23:49:28.261 DEBUG:teuthology.nuke:shortname: smithi176 2024-02-21T23:49:28.261 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-21T23:49:28.266 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi090.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-21_23:06:32-fs-wip-yuri10-testing-2024-02-21-1249-reef-distro-default-smithi/7570183', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-21 23:19:43.991167', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIsGbe8HkzhZE7QgpRJYvvS2/F3rk3+Sld7SbTDLMbrqSSFyoj3YfKfJ2KLL9DPsXS5F2GnhsB0d/msACzpTVlg='} 2024-02-21T23:49:28.284 DEBUG:teuthology.nuke:shortname: smithi178 2024-02-21T23:49:28.284 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-21T23:49:28.288 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi176.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-21_23:06:32-fs-wip-yuri10-testing-2024-02-21-1249-reef-distro-default-smithi/7570183', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-21 23:19:43.988364', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCoyhkEW5lZGHJcwJG3OfE/cNEzDU1Txk/PsJ5geG6e60QW55DpDVm/CbLAMmbRc/vh8k5rwNMjDeiHIQ5DMChg='} 2024-02-21T23:49:28.306 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi178.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-21_23:06:32-fs-wip-yuri10-testing-2024-02-21-1249-reef-distro-default-smithi/7570183', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-21 23:19:43.990009', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM5c7YRZvTsR1p2rGWIPkA7BaA2hCTX8hzMGAddvN5CwxwIwIzSkhSOOO3cjWtq3RpFs7pOCzOII6Pb1ofvzKN0='} 2024-02-21T23:49:28.317 INFO:teuthology.orchestra.console.smithi090:Power off 2024-02-21T23:49:28.317 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-21T23:49:28.343 DEBUG:teuthology.orchestra.console.smithi090:power off output: Chassis Power Control: Down/Off 2024-02-21T23:49:28.343 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-21T23:49:28.360 INFO:teuthology.orchestra.console.smithi176:Power off 2024-02-21T23:49:28.360 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-21T23:49:28.376 INFO:teuthology.orchestra.console.smithi178:Power off 2024-02-21T23:49:28.377 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-21T23:49:28.387 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on 2024-02-21T23:49:28.387 DEBUG:teuthology.orchestra.console.smithi176:power off output: Chassis Power Control: Down/Off 2024-02-21T23:49:28.387 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-21T23:49:28.399 DEBUG:teuthology.orchestra.console.smithi178:power off output: Chassis Power Control: Down/Off 2024-02-21T23:49:28.400 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-21T23:49:28.411 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2024-02-21T23:49:28.423 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is on 2024-02-21T23:49:32.388 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-21T23:49:32.411 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-21T23:49:32.424 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-21T23:49:32.502 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on 2024-02-21T23:49:32.523 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2024-02-21T23:49:32.534 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is on 2024-02-21T23:49:36.504 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-21T23:49:36.524 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-21T23:49:36.535 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-21T23:49:36.616 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is off 2024-02-21T23:49:36.637 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2024-02-21T23:49:36.646 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is off 2024-02-21T23:49:36.717 INFO:teuthology.orchestra.console.smithi090:Power off completed 2024-02-21T23:49:36.747 INFO:teuthology.orchestra.console.smithi178:Power off completed 2024-02-21T23:49:40.641 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-21T23:49:40.754 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is off 2024-02-21T23:49:40.855 INFO:teuthology.orchestra.console.smithi176:Power off completed 2024-02-21T23:49:40.960 INFO:teuthology.run:Summary data: description: fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} duration: 1413.084719657898 failure_reason: '''Filesystem'' object has no attribute ''run_ceph_cmd''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=9b27f320912d4810b1e4c3366f516ad1 status: fail success: false 2024-02-21T23:49:40.960 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-21T23:49:41.043 INFO:teuthology.run:FAIL